GPU acceleration not working in docker on Windows

[danielgv93]

Hi! Really cool project you have made! :)

I need to process several large videos, so I was trying out the hardware acceleration feature. I encountered an error while using Docker with the GPU profile you provided as an example. I'm not sure if the problem is with my configuration. My GPU is the GTX 1070 with the latest drivers, and I'm using Windows 11. The error itself is this:

PS C:\Users\..\Desktop\gopro-dashboard> docker run -it -v "$(pwd):/work" overlaydash/gopro-dashboard-overlay gopro-dashboard.py --config-dir=config/ --profile=nnvgpu --double-buffer --layout-xml=config/layout.xml videos/GX010024.MP4 render/docker.MP4
Starting gopro-dashboard version 0.115.0
ffmpeg version is 6.0.1
Using Python version 3.11.3 (main, May 11 2023, 23:30:05) [GCC 12.2.1 20220924]
Loading GoPro Data Track: 15,642,300 [ 14.9 MiB]  [ 61.2 MiB/s]
    Timer(GPMD - Called: 1, Total: 0.64113, Avg: 0.64113, Rate: 1.56)
>> Found GPS9
    Timer(extract GPS - Called: 1, Total: 1.52475, Avg: 1.52475, Rate: 0.66)
Timer(parsing - Called: 1, Total: 2.16602, Avg: 2.16602, Rate: 0.46)

NOTE: 10 GoPro GPS readings were mapped to 'NO_LOCK', for the following reasons:
* DOP > 10 -> 10

Generating overlay at Dimension(x=3840, y=2160)
Timeseries has 21140 data points
Processing....
FFMPEG Output is in /tmp/tmp4fqp7ibd.txt
Timelapse Factor = 1.000
Layout -> Include component 'gps_info' = True
Layout -> Include component 'gps-lock' = True
Layout -> Include component 'altitude' = True
Layout -> Include component 'moving_map' = True
Executing ['ffmpeg', '-y', '-hide_banner', '-loglevel', 'info', '-hwaccel', 'cuda', '-hwaccel_output_format', 'cuda', '-i', 'videos/GX010024.MP4', '-f', 'rawvideo', '-framerate', '10.0', '-s', '3840x2160', '-pix_fmt', 'rgba', '-i', '-', '-filter_complex', '[0:v]scale_cuda=format=yuv420p[mp4_stream];[1:v]format=yuva420p,hwupload[overlay_stream];[mp4_stream][overlay_stream]overlay_cuda', '-vcodec', 'h264_nvenc', '-rc:v', 'cbr', '-b:v', '40M', '-bf:v', '3', '-profile:v', 'main', '-spatial-aq', 'true', '-movflags', 'faststart', 'render/docker.MP4']
*** NOTE: Double Buffer mode is experimental. It is believed to work fine on Linux. Please raise issues if you see it working or not-working. Thanks ***
Render: 0 [  0%]  [  0.0/s] |                                                                          | ETA:  --:--:--Request for data at time Timeunit ms=0.0, before start of metadata, returning first item
Process Process-1:
Render: 1 [  0%]  [  2.4/s] |                                                                          | ETA:   2:27:46Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 153, in p_writer
    if not frame.write(writer):
           ^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 139, in write
    writer.write(self.memory)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 45, in write
    return super().write(__buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 153, in p_writer
    if not frame.write(writer):
           ^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 139, in write
    writer.write(self.memory)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 45, in write
    return super().write(__buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe

This is the gopro-dashboard/config/ffmpeg-profiles.json:

{
  "nvgpu": {
    "input": ["-hwaccel", "nvdec"],
    "output": ["-vcodec", "h264_nvenc", "-rc:v", "cbr", "-b:v", "50M", "-bf:v", "3", "-profile:v", "high", "-spatial-aq", "true"]
  },
  "nnvgpu": {
    "input": ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda" ],
    "filter": "[0:v]scale_cuda=format=yuv420p[mp4_stream];[1:v]format=yuva420p,hwupload[overlay_stream];[mp4_stream][overlay_stream]overlay_cuda",
    "output": ["-vcodec", "h264_nvenc", "-rc:v", "cbr", "-b:v", "40M", "-bf:v", "3", "-profile:v", "main", "-spatial-aq", "true", "-movflags", "faststart"]
  },

  "low-fr": {
    "input": [],
    "output": ["-r", "10"]
  }
}

[time4tea]

Hi! Thanks for your interest in the software!

There's a lot going on here... and getting the best performance is a bit tricky.

You might find that using --show-ffmpeg gives you more information.

However.. the docker image does not contain the gpu extensions, and I'm not sure what docker magic is required to expose the gpu to the container, and that goes double for windows. I am using a relatively old base layer for docker, so I could check if this situation has changed.

On Windows natively, which can support gpus, double buffer mode isn't supported, as Windows does not support the sharing of a file handle between two processes, which is trivial in Unix. I think there is a way to rewrite the double buffer mode so that it can work, but it is a big job. Other people have reported that native Windows is quite a bit quicker than using docker on Windows (for this software) - maybe that is worth trying..

So I'm sorry I don't have an immediate solution for you...

[danielgv93]

Thanks for your time and quick response. The docker on Windows using only the --double-buffer parameter works fine. It seems that it is only the ffmpeg profile that fails. I leave the log with the --show-ffmpeg parameter:

PS C:\Users\...\Desktop\gopro-dashboard> docker run -it -v "$(pwd):/work" overlaydash/gopro-dashboard-overlay gopro-dashboard.py --show-ffmpeg --config-dir=config/ --profile=nnvgpu --double-buffer --layout-xml=config/layout.xml videos/GX010024.MP4 render/docker.MP4
Starting gopro-dashboard version 0.115.0
Running ['ffmpeg', '-version']
Running ['ffmpeg', '-v', 'quiet', '-codecs']
Running ['ffmpeg', '-version']
ffmpeg version is 6.0.1
Using Python version 3.11.3 (main, May 11 2023, 23:30:05) [GCC 12.2.1 20220924]
Running ['-hide_banner', '-y', '-i', PosixPath('videos/GX010024.MP4'), '-codec', 'copy', '-map', '0:3', '-f', 'rawvideo', '-']
Loading GoPro Data Track: 15,642,300 [ 14.9 MiB]  [ 73.6 MiB/s]
    Timer(GPMD - Called: 1, Total: 0.55112, Avg: 0.55112, Rate: 1.81)
>> Found GPS9
    Timer(extract GPS - Called: 1, Total: 1.43386, Avg: 1.43386, Rate: 0.70)
Timer(parsing - Called: 1, Total: 1.98510, Avg: 1.98510, Rate: 0.50)

NOTE: 10 GoPro GPS readings were mapped to 'NO_LOCK', for the following reasons:
* DOP > 10 -> 10

Generating overlay at Dimension(x=3840, y=2160)
Timeseries has 21140 data points
Processing....
Timelapse Factor = 1.000
Layout -> Include component 'gps_info' = True
Layout -> Include component 'gps-lock' = True
Layout -> Include component 'altitude' = True
Layout -> Include component 'moving_map' = True
Executing ['ffmpeg', '-y', '-hide_banner', '-loglevel', 'info', '-hwaccel', 'cuda', '-hwaccel_output_format', 'cuda', '-i', 'videos/GX010024.MP4', '-f', 'rawvideo', '-framerate', '10.0', '-s', '3840x2160', '-pix_fmt', 'rgba', '-i', '-', '-filter_complex', '[0:v]scale_cuda=format=yuv420p[mp4_stream];[1:v]format=yuva420p,hwupload[overlay_stream];[mp4_stream][overlay_stream]overlay_cuda', '-vcodec', 'h264_nvenc', '-rc:v', 'cbr', '-b:v', '40M', '-bf:v', '3', '-profile:v', 'main', '-spatial-aq', 'true', '-movflags', 'faststart', 'render/docker.MP4']
*** NOTE: Double Buffer mode is experimental. It is believed to work fine on Linux. Please raise issues if you see it working or not-working. Thanks ***
Unrecognized option 'rc:v'.
Error splitting the argument list: Option not found
Render: 0 [  0%]  [  0.0/s] |                                                                          | ETA:  --:--:--Request for data at time Timeunit ms=0.0, before start of metadata, returning first item
Process Process-1:
Render: 1 [  0%]  [  1.8/s] |                                                                          | ETA:   3:11:04Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 153, in p_writer
    if not frame.write(writer):
           ^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 139, in write
    writer.write(self.memory)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 45, in write
    return super().write(__buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 153, in p_writer
    if not frame.write(writer):
           ^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 139, in write
    writer.write(self.memory)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 45, in write
    return super().write(__buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe

There seems to be some incompatibility with the ffmpeg options. Removing the conficting options, the log looks like this:

PS C:\Users\...\Desktop\gopro-dashboard> docker run -it -v "$(pwd):/work" overlaydash/gopro-dashboard-overlay gopro-dashboard.py --show-ffmpeg --config-dir=config/ --profile=nnvgpu --double-buffer --layout-xml=config/layout.xml videos/GX010024.MP4 render/docker.MP4
Starting gopro-dashboard version 0.115.0
Running ['ffmpeg', '-version']
Running ['ffmpeg', '-v', 'quiet', '-codecs']
Running ['ffmpeg', '-version']
ffmpeg version is 6.0.1
Using Python version 3.11.3 (main, May 11 2023, 23:30:05) [GCC 12.2.1 20220924]
Running ['-hide_banner', '-y', '-i', PosixPath('videos/GX010024.MP4'), '-codec', 'copy', '-map', '0:3', '-f', 'rawvideo', '-']
Loading GoPro Data Track: 15,642,300 [ 14.9 MiB]  [ 81.2 MiB/s]
    Timer(GPMD - Called: 1, Total: 0.55540, Avg: 0.55540, Rate: 1.80)
>> Found GPS9
    Timer(extract GPS - Called: 1, Total: 1.41331, Avg: 1.41331, Rate: 0.71)
Timer(parsing - Called: 1, Total: 1.96881, Avg: 1.96881, Rate: 0.51)

NOTE: 10 GoPro GPS readings were mapped to 'NO_LOCK', for the following reasons:
* DOP > 10 -> 10

Generating overlay at Dimension(x=3840, y=2160)
Timeseries has 21140 data points
Processing....
Timelapse Factor = 1.000
Layout -> Include component 'gps_info' = True
Layout -> Include component 'gps-lock' = True
Layout -> Include component 'altitude' = True
Layout -> Include component 'moving_map' = True
Executing ['ffmpeg', '-y', '-hide_banner', '-loglevel', 'info', '-hwaccel', 'cuda', '-hwaccel_output_format', 'cuda', '-i', 'videos/GX010024.MP4', '-f', 'rawvideo', '-framerate', '10.0', '-s', '3840x2160', '-pix_fmt', 'rgba', '-i', '-', '-filter_complex', '[0:v][1:v]overlay', '-vcodec', 'h264_nvenc', '-b:v', '40M', '-bf:v', '3', '-profile:v', 'main', '-movflags', 'faststart', 'render/docker.MP4']
*** NOTE: Double Buffer mode is experimental. It is believed to work fine on Linux. Please raise issues if you see it working or not-working. Thanks ***
Render: 0 [  0%]  [  0.0/s] |                                                                          | ETA:  --:--:--Request for data at time Timeunit ms=0.0, before start of metadata, returning first item
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'videos/GX010024.MP4':
  Metadata:
    major_brand     : mp41
    minor_version   : 538120216
    compatible_brands: mp41
    creation_time   : 2023-12-05T15:20:43.000000Z
    firmware        : H22.01.02.20.00
  Duration: 00:35:14.11, start: 0.000000, bitrate: 45256 kb/s
  Stream #0:0[0x1](eng): Video: hevc (Main) (hvc1 / 0x31637668), yuvj420p(pc, bt709), 3840x2160 [SAR 1:1 DAR 16:9], 44990 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
    Metadata:
      creation_time   : 2023-12-05T15:20:43.000000Z
      handler_name    : GoPro H.265
      vendor_id       : [0][0][0][0]
      encoder         : GoPro H.265 encoder
      timecode        : 16:19:44:23
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 189 kb/s (default)
    Metadata:
      creation_time   : 2023-12-05T15:20:43.000000Z
      handler_name    : GoPro AAC
      vendor_id       : [0][0][0][0]
      timecode        : 16:19:44:23
  Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74) (default)
    Metadata:
      creation_time   : 2023-12-05T15:20:43.000000Z
      handler_name    : GoPro TCD
      timecode        : 16:19:44:23
  Stream #0:3[0x4](eng): Data: bin_data (gpmd / 0x646D7067), 59 kb/s (default)
    Metadata:
      creation_time   : 2023-12-05T15:20:43.000000Z
      handler_name    : GoPro MET
Render: 1 [  0%]  [  1.8/s] |                                                                          | ETA:   3:18:26Input NVIDIA/nvidia-docker#1, rawvideo, from 'fd:':
  Duration: N/A, start: 0.000000, bitrate: 2654208 kb/s
  Stream NVIDIA/nvidia-docker#1:0: Video: rawvideo (RGBA / 0x41424752), rgba, 3840x2160, 2654208 kb/s, 10 tbr, 10 tbn
[vost#0:0 @ 0x7fa31845c8c0] Unknown encoder 'h264_nvenc'
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 153, in p_writer
    if not frame.write(writer):
           ^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 139, in write
    writer.write(self.memory)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 45, in write
    return super().write(__buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
Process Process-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 153, in p_writer
    if not frame.write(writer):
           ^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 139, in write
    writer.write(self.memory)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 45, in write
    return super().write(__buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe

[time4tea]

hi - yes - its as I said - docker version of ffmpeg doesn't support GPU.

You can see this from the error:

Unknown encoder 'h264_nvenc'

In any case, the actual docker container doesn't support accessing the GPU in the container, as there are some extra things that need to be done to support this.

Double buffer mode will work ok in docker on windows (as it is essentially running on unix) - its only windows native where there is a problem.

You can try to install using Windows native python, this will enable the GPU. If your CPU is fast, this will probably give a good speed boost. (But double buffering won't work)

[danielgv93]

Thank you very much! really appreciate the help

[time4tea]

I do appreciate and agree that the error messages are hard to decipher! - It is a lot of work to interpret all the various error codes that all the things can emit.

[time4tea]

I think perhaps I could make the docker image differently so that it will support GPUs.

[danielgv93]

It would be great! Thank you

[time4tea]

This is now working in version 0.116.0 - there are some docs in docs/docker.md

[danielgv93]

First of all, thank you for your effort!!

Since the NVIDIA Container Toolkit is not available on Windows, it doesn't seem as straightforward as on Linux. I will check how to run it.

As far as I have tried, I got this:

On PowerShell:

PS C:\Users\...\Desktop\gopro-apps> docker run --gpus all -it -v "$(pwd):/work" overlaydash/gopro-dashboard-overlay gopro-dashboard.py --config-dir=config --profile=nnvgpu --layout-xml=config/layout.xml --show-ffmpeg videos/GX010026.MP4 result/output.MP4
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/55da83471a229515999dd6e0c3e36079330844a108ea0af8998868a00e743cbb/merged/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1: file exists: unknown.

I tried on Wsl Terminal but the same error appeared:

...@DESKTOP-4TK1A0V:/mnt/c/Users/.../Desktop/gopro-apps$ sudo docker run --gpus all -it -v "$(pwd):/work" overlaydash/gopro-dashboard-overlay gopro-dashboard.py --config-dir=config --profile=nnvgpu --double-buffer --layout-xml=config/layout.xml --show-ffmpeg videos/GX010026.MP4 result/output.MP4
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/818c23b0e6027c2d73510b7c7d66be15fd08a5d40cac12f5335d7f7d638018c1/merged/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1: file exists: unknown.
ERRO[0000] error waiting for container:

[time4tea]

It should be supported for Windows - there is some information here: https://www.docker.com/blog/wsl-2-gpu-support-for-docker-desktop-on-nvidia-gpus/

[danielgv93]

I do not think it is an Nvidia issue, as I have run several Nvidia tests like this without any problems.

PS C:\Users\...\Desktop\gopro-apps> docker run -it --gpus=all --rm nvidia/cuda:12.3.1-base-ubuntu22.04 nvidia-smi
Sat Jan  6 00:14:59 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.36                 Driver Version: 546.33       CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1070        On  | 00000000:01:00.0  On |                  N/A |
| 31%   46C    P0              38W / 180W |   1769MiB /  8192MiB |      9%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

[time4tea]

OK... bit more debugging required. The image does work Linux, so there is a bit of different behaviour here. So far I have found that this gets a bit futher... I'll put this into the actual software once I've got a version that can be shown to work.

I found some info here: https://github.com/NVIDIA/nvidia-container-toolkit/issues/289

Make a local Dockerfile.txt - I used notepad, so it only makes txt files!

FROM overlaydash/gopro-dashboard-overlay:latest

RUN rm -rf /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1 /usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/x86_64-linux-gnu/libcudadebugger.so.1

 docker build -t temp -f .\Dockerfile.txt .

Output (I removed some unrelated lines)

PS G:\james\transfer\james>  docker run --gpus all -it -v "$(pwd):/work" xx gopro-dashboard.py --profile=nnvgpu --show-ffmpeg GH010207.MP4 output.MP4
Starting gopro-dashboard version 0.122.0
Running ['ffmpeg', '-version']
Running ['ffmpeg', '-v', 'quiet', '-codecs']
Running ['ffmpeg', '-version']
ffmpeg version is 6.0-6ubuntu1
Using Python version 3.11.6 (main, Oct  8 2023, 05:06:43) [GCC 13.2.0]
Running ['-hide_banner', '-y', '-i', PosixPath('GH010207.MP4'), '-codec', 'copy', '-map', '0:3', '-f', 'rawvideo', '-']
Loading GoPro Data Track: 3,409,836 [  3.3 MiB]  [ 68.6 MiB/s]
    Timer(GPMD - Called: 1, Total: 0.06220, Avg: 0.06220, Rate: 16.08)
>> Found GPS5
    Timer(extract GPS - Called: 1, Total: 0.50802, Avg: 0.50802, Rate: 1.97)
Timer(parsing - Called: 1, Total: 0.57033, Avg: 0.57033, Rate: 1.75)
Generating overlay at Dimension(x=2704, y=1520)
Timeseries has 9662 data points
Processing....
Using *built-in* profile: nnvgpu
Timelapse Factor = 1.000
Executing ['ffmpeg', '-y', '-hide_banner', '-loglevel', 'info', '-hwaccel', 'cuda', '-hwaccel_output_format', 'cuda', '-i', 'GH010207.MP4', '-f', 'rawvideo', '-framerate', '10.0', '-s', '2704x1520', '-pix_fmt', 'rgba', '-i', '-', '-filter_complex', '[0:v]scale_cuda=format=yuv420p[mp4_stream];[1:v]format=yuva420p,hwupload[overlay_stream];[mp4_stream][overlay_stream]overlay_cuda', '-vcodec', 'h264_nvenc', '-rc:v', 'cbr', '-b:v', '25M', '-bf:v', '3', '-profile:v', 'main', '-spatial-aq', 'true', '-movflags', 'faststart', 'output.MP4']
Render: 0 [  0%]  [  0.0/s] |                                                                                                                                                                                                                | ETA:  --:--:--[mov,mp4,m4a,3gp,3g2,mj2 @ 0x555f25e30200] All samples in data stream index:id [4:5] have zero duration, stream set to be discarded by default. Override using AVStream->discard or -discard for ffmpeg command.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'GH010207.MP4':
  Metadata:
    major_brand     : mp41
    minor_version   : 538120216
    compatible_brands: mp41
    creation_time   : 2022-05-29T11:22:39.000000Z
    location        : +51.3355-000.4085/
    location-eng    : +51.3355-000.4085/
    firmware        : HD9.01.01.70.00
  Duration: 00:08:51.54, start: 0.000000, bitrate: 60270 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709, progressive), 2704x1520 [SAR 1:1 DAR 169:95], 60002 kb/s, 59.94 fps, 59.94 tbr, 60k tbn (default)
    Metadata:
      creation_time   : 2022-05-29T11:22:39.000000Z
      handler_name    : GoPro AVC
      vendor_id       : [0][0][0][0]
      encoder         : GoPro AVC encoder
      timecode        : 11:21:59:01
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 189 kb/s (default)
    Metadata:
      creation_time   : 2022-05-29T11:22:39.000000Z
      handler_name    : GoPro AAC
      vendor_id       : [0][0][0][0]
      timecode        : 11:21:59:01
  Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74) (default)
    Metadata:
      creation_time   : 2022-05-29T11:22:39.000000Z
      handler_name    : GoPro TCD
      timecode        : 11:21:59:01
  Stream #0:3[0x4](eng): Data: bin_data (gpmd / 0x646D7067), 51 kb/s (default)
    Metadata:
      creation_time   : 2022-05-29T11:22:39.000000Z
      handler_name    : GoPro MET
  Stream #0:4[0x5](eng): Data: none (fdsc / 0x63736466), 13 kb/s (default)
    Metadata:
      creation_time   : 2022-05-29T11:22:39.000000Z
      handler_name    : GoPro SOS
Input NVIDIA/nvidia-docker#1, rawvideo, from 'fd:':
  Duration: N/A, start: 0.000000, bitrate: 1315225 kb/s
  Stream NVIDIA/nvidia-docker#1:0: Video: rawvideo (RGBA / 0x41424752), rgba, 2704x1520, 1315225 kb/s, 10 tbr, 10 tbn
Render: 1 [  0%]  [  1.8/s] |                                                                                                                                                                                                                | ETA:   0:48:12Stream mapping:
  Stream #0:0 (h264) -> scale_cuda:default (graph 0)
  Stream NVIDIA/nvidia-docker#1:0 (rawvideo) -> format:default (graph 0)
  overlay_cuda:default (graph 0) -> Stream #0:0 (h264_nvenc)
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
[h264 @ 0x555f26f85a80] decoder->cvdl->cuvidGetDecoderCaps(&caps) failed -> CUDA_ERROR_INVALID_DEVICE: invalid device ordinal
[h264 @ 0x555f26f85a80] Using more than 32 (34) decode surfaces might cause nvdec to fail.
[h264 @ 0x555f26f85a80] Try lowering the amount of threads. Using 16 right now.
[h264 @ 0x555f26f85a80] Failed setup for format cuda: hwaccel initialisation returned error.
Impossible to convert between the formats supported by the filter 'graph 0 input from stream 0:0' and the filter 'auto_scale_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream NVIDIA/nvidia-docker#1:0
[aac @ 0x555f262d0740] Qavg: 61490.887
[aac @ 0x555f262d0740] 2 frames left in the queue on closing
Render: 2 [  0%]  [  1.9/s] |                                                                                                                                                                                                                | ETA:   0:45:49Conversion failed!
Waiting for ffmpeg to complete...
FFMPEG Exited with status code: 1
Timer(drawing frames - Called: 3, Total: 1.06213, Avg: 0.35404, Rate: 2.82)
Traceback (most recent call last):
  File "/venv/bin/gopro-dashboard.py", line 366, in <module>
    with ffmpeg.generate() as writer:
  File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/venv/lib/python3.11/site-packages/gopro_overlay/ffmpeg_overlay.py", line 107, in generate
    yield from self.exe.execute(
  File "/venv/lib/python3.11/site-packages/gopro_overlay/ffmpeg.py", line 68, in execute
    yield from execution.execute([self._path(), *args])
  File "/venv/lib/python3.11/site-packages/gopro_overlay/execution.py", line 46, in execute
    raise IOError(f"Process {cmd[0]} failed") from None
OSError: Process ffmpeg failed

So... it seems that for wahtever reason, I can't select the correct graphics device, or perhaps I need to fix the thread count problem - I've seen this before, and a fix is in our docs. However, I don't know if this is an error just for me, or will affect everyone. I'll iterate on the ffmpeg profile....

[time4tea]

Starting the container to get a prompt

docker run --gpus all -it -v "$(pwd):/work" --entrypoint /bin/bash xx

and then getting ffmpeg to show the gpus

ffmpeg -vsync 0 -i GH010207.MP4 -c:v h264_nvenc -gpu list -f null -

Returns

[h264_nvenc @ 0x55795bf18700] [ GPU #0 - < NVIDIA GeForce RTX 3060 > has Compute SM 8.6 ]

So I think that the invalid device ordinal might be a red herring.....

[time4tea]

Not quite sure what else to look into here? All the nvidia tools report that things should work, but ffmpeg doesn't like it, when running in a container, but only on windows.