GPX <-> Video offset

[dkrizic]

Hi, I started using your software and would like to share my experience and problems. My typical approach when cycling is that is have 3 devices with me:

• Apple Watch -> Delivers GPX and FIT via HealthFit App
• Garmin Edge -> Delivers GPX
• GoPro Hero 11 -> Embed GPX in the videos

I found out that the GPS of the GoPro is basically useless. It needs a long time to lock and looses signal on many occurrences where especially the Apple Watch Ultra works perfectly. While the ride I activate the cameras many times, it is not running permanently. So after the ride what I have is:

• A GPX or a FIT file that covers the whole ride, usually starts earlier and ends after the first/last GoPro recording
• Many GX*.MP4 files, all in the timeframe of the GPX/FIT.

I would like to have a single concatenated video that shows the whole Journey file. I am writing some shell scripts in order to automate my steps. My current approach is:

• Iterate over all MP4 files
  • Fix the timestamp (see below)
  • Create a dashboard version of the video with --gpx=.... --gpx-only and --video-time-start file-modified
  • Concatenate all generated videos

Currently I have two problems:

• I am providing the full GPX track, but the journey only contains the fraction that belongs to the timestamp of the video -> That should be configurable
• The track has a time offset of about a minute, not hour (usually timezone or so)

The way how I timestamp the files is basically the following. ffprobe gives some longish output that contains the timestamp of creation, like this

ffprobe version 6.1.1 Copyright (c) 2007-2023 the FFmpeg developers
  built with Apple clang version 15.0.0 (clang-1500.1.0.2.5)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/6.1.1_4 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopenvino --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'GX010051.MP4':
  Metadata:
    major_brand     : mp41
    minor_version   : 538120216
    compatible_brands: mp41
    creation_time   : 2024-03-02T11:51:29.000000Z
    firmware        : H22.01.02.20.00
  Duration: 00:01:31.44, start: 0.000000, bitrate: 119957 kb/s
  Stream #0:0[0x1](eng): Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(pc, bt709), 3840x2160 [SAR 1:1 DAR 16:9], 119684 kb/s, 50 fps, 50 tbr, 90k tbn (default)
    Metadata:
      creation_time   : 2024-03-02T11:51:29.000000Z
      handler_name    : GoPro H.265
      vendor_id       : [0][0][0][0]
      encoder         : GoPro H.265 encoder
      timecode        : 12:51:29:17
    Side data:
      displaymatrix: rotation of -180.00 degrees
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 189 kb/s (default)
    Metadata:
      creation_time   : 2024-03-02T11:51:29.000000Z
      handler_name    : GoPro AAC
      vendor_id       : [0][0][0][0]
      timecode        : 12:51:29:17
  Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74) (default)
    Metadata:
      creation_time   : 2024-03-02T11:51:29.000000Z
      handler_name    : GoPro TCD
      timecode        : 12:51:29:17
  Stream #0:3[0x4](eng): Data: bin_data (gpmd / 0x646D7067), 62 kb/s (default)
    Metadata:
      creation_time   : 2024-03-02T11:51:29.000000Z
      handler_name    : GoPro MET
Unsupported codec with id 0 for input stream 2
Unsupported codec with id 98314 for input stream 3

Then I can touch the file accordingly

touch -d 2024-03-02T11:51:29.000000Z -m GX010051.MP4

which gives me the correct date and time of the file:

ls -lF GX010051.MP4
.rw-------@ 1.4G dkrizic  2 Mar 12:51 GX010051.MP4

(I am one hour away from Zulu, so the hour difference is fine).

Additionally it would make sense to have a feature that shows the journey of the full length of the GPX/FIT.

[dkrizic]

But reading my own stuff, I probably need to ensure that the timestamp has the right formats, maybe the offset comes from missing/wrong seconds.

[dkrizic]

I just checked, the time is perfect:

s -la --time-style=full-iso GX010051.MP4
.rw-------@ 1.4G dkrizic 2024-03-02 12:51:29.000000000 +0100 GX010051.MP4

[dkrizic]

The time offset is 40 seconds, where the GPX signal is ahead in time. So here it shows me boing at the bend at 12:49:20

But I actually arrive at 12:50:01

[time4tea]

Hi. Thanks for your interest in the project.

Ultimately, this is a tricky problem... tbh usually the sync of a hero 11 is not too bad, but in any case, syncing things is hard.

We have lots of time sources. Gps from the gopro, gps from the gpx source and also file metadata.

In the case of gopro files generally, I've found that file metadata is the least reliable.

I have two proposed options that are sort of easy to do

1. Don't clip gpx files to the video length
2. Use a better way of determining the sync point by finding a point somewhere in the video, rather than just at the start.

This would be easier, but gpx files are usually 1 data point per second.

I think these would allow you to do most of the things you wanted.. Cheers

James

[jburnhams]

I'd be interested in this - "Don't clip gpx files to the video length" - so I can process short video clips from a longer journey and have the journey_map show the entire gpx (with my current position marked), not just the portion of the track that the video covers. Is this currently an option, or would it be a new feature?

[time4tea]

This would be a new feature.. its something that I think could be quite useful.

[jburnhams]

Your code is great, really well structured and easy to follow - I think I can see how to change to support so I'll send a PR with my changes (tho I doubt they'll be good enough to merge as is!)

[jamesr-home]

That is very kind, although tbh a lot of it is pretty rough - but i don't get to spend much time on it.

The main problem I think at the moment is that there is no real differentiation between the start and end of the movie, and the start and end of the "gopro/gpx" data. Need to introduce something so that this difference is specific. Unfortunately this bit of the code hasn't had much love since the early days....

[jamesr-home]

The time offset is 40 seconds, where the GPX signal is ahead in time. So here it shows me boing at the bend at 12:49:20

But I actually arrive at 12:50:01

As far as i understand it, It is sort of impossible for the GPX and GoPro to be out of sync by this much, for the actual GPS data. As GPS basically relies on very accurate times. Indeed the US government promises that the time will be accurate within 30ns (nanoseconds).

That having been said... if the gopro isn't locked, the time will be wrong, and the file metadata is basically always wrong. (unless you have the gopro labs "file metadata gps sync" firmaware patch) - because the gopro file metadata is updated from the internal clock, not from GPS time.

It is of course quite possible that the time processing in thei sosftware is a bit wrong - and the file access time code is not used so much, so it could be an error there.

[dkrizic]

I have an event with about 10 videos and I observed the following:

• The first 3 videos have no GPS at all, the GoPro obviously did not lock
• The remaining 7 videos have GPS

So in order to have a somewhat reliable source of when the actual video start I use that feature that I mentioned, I ask ffprobe and use the creation_time, this creation_time is always available, even in files where the GoPro never locked. The interesting part is that there always seems to be an offset of 40 seconds.

Here is the Shell script that I use for doing everything: https://github.com/dkrizic/overlay/blob/main/overlay. This shell script needs a directory as parameter. I just throw in a FIT file and all my GoPro videos from that event/journey and it processes each file and merges then all files.

In line 43 I will add 40s to the video start and take a look at the result and report.

[time4tea]

This does make sense - the file metadata (from ffprobe) is the one that doesn't come from GPS - its from an internal clock.. so it will drift over time, unless it is regularly reset - you can do this with the app, or manually, but tbh i find this to be a massive pain.

You can add a feature to the gopro firmware - follow the instructions at: https://gopro.github.io/labs/control/gpssync/ - which will update the file metadata according to the GPS time, rather than the internal clock. I think (untested) that this will work even when the GPS time is obtained part way through the recording. I think a more-or-less accurate time can be obtained even before the GPS locks, so this might be useful.

[dkrizic]

Thank you for the link. I was assuming that every time the GoPro has GPS, that the internal clock will be synched. That is what most devices do. But I will definitively activate this feature. Even if the GPS track is not very reliable, once it has GPS it can synchronise the clock precisely.

[jburnhams]

I've raised a couple of PRs - features work for me but have only done very limited testing...

[dkrizic]

Looks good, so the PR implement the ability to have the full journey based on the GPX/FIT and to read out the timestamp directly from the MP4. Great.

[dkrizic]

BTW, after applying an offset of -40 seconds to the video file timestamp, the sync is now perfect:

I will apply the above mentioned lab feature to sync the clock via GPS so that should not be the issue anymore. But my script is now able to handle an offset as well.

[jburnhams]

Thanks again for a great bit of software, here's a video I've made with it https://youtu.be/ip7nMh13nww?si=COCs-XWXawMIYZdH

[dkrizic]

Thanks for the tip with the GoPro Labs and the possibility to sync the clock using GPS. After sync the offset is gone and it matches perfectly. Will there be a version soon that can display the whole GPX track even if only a fraction is processed in the video?