subprocess timeout from gpmd run

[rpoxo]

root@x570d4i-2t:/home/rpoxo/gopro-dashboard-overlay/docker# docker logs gopro-overlay --follow
Starting gopro-dashboard version 0.108.0
ffmpeg version is 6.0
Using Python version 3.11.3 (main, May 11 2023, 23:30:05) [GCC 12.2.1 20220924]
Traceback (most recent call last):
  File "/venv/lib/python3.11/site-packages/gopro_overlay/loading.py", line 51, in load
    recording.load_gpmd(),
    ^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/gopro_overlay/ffmpeg_gopro.py", line 212, in load_gpmd
    result = self.ffmpeg.run(cmd, capture_output=True, timeout=10)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/gopro_overlay/ffmpeg.py", line 55, in run
    return run(args_, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/gopro_overlay/process.py", line 5, in run
    return subprocess.run(cmd, check=True, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/subprocess.py", line 1207, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/subprocess.py", line 2076, in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
  File "/usr/local/lib/python3.11/subprocess.py", line 1251, in _check_timeout
    raise TimeoutExpired(
subprocess.TimeoutExpired: Command '['ffmpeg', '-hide_banner', '-y', '-i', PosixPath('input.mp4'), '-codec', 'copy', '-map', '0:2', '-f', 'rawvideo', '-']' timed out after 10 seconds
input.mp4 appears to be located on a slow device. Please ensure both input and output files are on fast disks

Is it possible to increase timeout for gpmd ffmpeg subprocess run, or make it configurable from env variables? I'm using separate machine for applying overlay, and directory with source file mounted from fast ssd(barely ~5% activity during actual render) via cifs over gigabit lan(~10% activity during actual render). Source file is concatenated ~150G 4k 3hr footage, so i suspect the filesize is the issue here. Increasing timeout to 20-30 seconds prevents error, and scripts doing their job just fine.

[time4tea]

Hi - Thanks for using the software!

I'll add a way to configure the timeout, or detect the data rate or something. However.. networking and in particular SMB/CIFS settings can have a huge impact on performance. Out of curiosity, when copying a large file to and from the remote server, are you able to saturate the link? You should be able to get almost 100% of the link speed if everything is working OK.

Also, I'd be really interested to know what you are using the software for, if you are happy to share - no worries at all if not!

[time4tea]

An extended timeout is available in v0.111.0 - should be on PyPI now..

[rpoxo]

saturating link

Yes, if i'm copying big files, atop reporting ~98% usage, ~975Mbps, rsync reports ~110 MB/s average, which i believe should on par with linear read for regular hdd's.

And here's usage patten when launching&running container. I haven't tested it yet on 10G link, but i believe result would be similar due to low IOPS over networked storage.

use case

Long ass boring gopro videos of me traveling on bike :) My friends were interested in looking at speed/cadence/heartrate stats during hard sections, aswell as map track on video, rather than compare tracks from strava, and then guessing where it is on video. Before i've been using VIRB edit for same purposes, but it's not supported anymore, and manually merging data track with video were pain, aswell as drifting over time longer than 5 minutes were less than desirable. My current process much easier, i just do 2 commands and can do something else meanwhile:

1. concatenade gopro videos into one with copying gpmd data from video. ffmpeg.exe -f concat -i mylist.txt -c copy -map 0:v -map 0:a -map 0:3 -copy_unknown -tag:2 gpmd concatenated.mp4
2. run gopro-dashboard-overlay and let it do it's thing: docker run -it -d --rm --name gopro-overlay -v "$(pwd):/work" dev/gopro-dashboard-overlay:0.108.1 gopro-dashboard.py --units-speed kph --units-altitude meter --units-distance km --gpx Afternoon_Gravel_Ride.gpx concatenated.mp4 concatenated_.mp4

I haven't gotten around hardware acceleration yet, through i do know it should be possible via vaapi and passing rendering device to docker container. As software encoding even on veryfast preset takes ~4 hours for 3h30m 4k@30fps 100mbps on 5950x with 75% cores utilization.

[time4tea]

Thanks for this info! There is a fix for this in v0.111.0, improved hopefully in v0.112.0 - now timeout is longer, and it also shows speed of loading data. Note that the speed indicated is for the data track, not necessarily the speed of reading the underlying file.

Some minor thoughts... Docker can give worse performance than running natively. ffmpeg in docker doesn't AFAIK have GPU extensiions, and may not be as optimised for your particular machine. Have you tried gopro_join.py ? I'd be interested if it works for you.

[time4tea]

Hi @rpoxo was this better?