Error thrown when running docker compose for Plex container

bigverm23 commented 4 years ago

The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

Also, before reporting a new issue, please make sure that:

You read carefully the documentation and frequently asked questions.
You searched for a similar issue and this is not a duplicate of an existing one.
This issue is not related to NGC, otherwise, please use the devtalk forums instead.
You went through the troubleshooting steps.

1. Issue or feature description

ERROR: for plex Cannot start service plex: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/7a5db13e7762a66cc95bc070c9fb3be25c72ef6df42250d0abf433029c4f0286/log.json: no such file or directory): exec: "nvidia-container-runtime": executable file not found in $PATH: unknown

2. Steps to reproduce the issue

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit sudo systemctl restart docker

3. Information to attach (optional if deemed irrelevant)

[x] Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info `-- WARNING, the following logs are for debugging purposes only --

I0506 20:06:53.423130 416509 nvc.c:281] initializing library context (version=1.0.7, build=b71f87c04b8eca8a16bf60995506c35c937347d9) I0506 20:06:53.423195 416509 nvc.c:255] using root / I0506 20:06:53.423206 416509 nvc.c:256] using ldcache /etc/ld.so.cache I0506 20:06:53.423214 416509 nvc.c:257] using unprivileged user 1000:1000 W0506 20:06:53.424703 416510 nvc.c:186] failed to set inheritable capabilities W0506 20:06:53.424767 416510 nvc.c:187] skipping kernel modules load due to failure I0506 20:06:53.425112 416511 driver.c:133] starting driver service I0506 20:06:53.447177 416509 nvc_info.c:438] requesting driver information with '' I0506 20:06:53.447755 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvoptix.so.440.64 I0506 20:06:53.447857 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.440.64 I0506 20:06:53.447921 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.440.64 I0506 20:06:53.447993 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.440.64 I0506 20:06:53.448083 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.440.64 I0506 20:06:53.448174 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.440.64 I0506 20:06:53.448238 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.440.64 I0506 20:06:53.448327 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ifr.so.440.64 I0506 20:06:53.448420 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.440.64 I0506 20:06:53.448482 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.440.64 I0506 20:06:53.448543 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.440.64 I0506 20:06:53.448604 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.440.64 I0506 20:06:53.448690 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.440.64 I0506 20:06:53.448753 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.440.64 I0506 20:06:53.448843 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.440.64 I0506 20:06:53.448912 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.440.64 I0506 20:06:53.448979 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.440.64 I0506 20:06:53.449059 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cbl.so.440.64 I0506 20:06:53.449124 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvcuvid.so.440.64 I0506 20:06:53.449656 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libcuda.so.440.64 I0506 20:06:53.450090 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.440.64 I0506 20:06:53.450153 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.440.64 I0506 20:06:53.450220 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.440.64 I0506 20:06:53.450286 416509 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.440.64 I0506 20:06:53.450389 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-tls.so.440.64 I0506 20:06:53.450447 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-ptxjitcompiler.so.440.64 I0506 20:06:53.450561 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-opticalflow.so.440.64 I0506 20:06:53.450658 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-opencl.so.440.64 I0506 20:06:53.450723 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-ml.so.440.64 I0506 20:06:53.450814 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-ifr.so.440.64 I0506 20:06:53.450903 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-glvkspirv.so.440.64 I0506 20:06:53.450969 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-glsi.so.440.64 I0506 20:06:53.451031 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-glcore.so.440.64 I0506 20:06:53.451093 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-fbc.so.440.64 I0506 20:06:53.451183 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-fatbinaryloader.so.440.64 I0506 20:06:53.451265 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-encode.so.440.64 I0506 20:06:53.451368 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-eglcore.so.440.64 I0506 20:06:53.451433 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvidia-compiler.so.440.64 I0506 20:06:53.451497 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libnvcuvid.so.440.64 I0506 20:06:53.451603 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libcuda.so.440.64 I0506 20:06:53.451708 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libGLX_nvidia.so.440.64 I0506 20:06:53.451769 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libGLESv2_nvidia.so.440.64 I0506 20:06:53.451828 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libGLESv1_CM_nvidia.so.440.64 I0506 20:06:53.451885 416509 nvc_info.c:152] selecting /usr/lib/i386-linux-gnu/libEGL_nvidia.so.440.64 W0506 20:06:53.451920 416509 nvc_info.c:303] missing library libvdpau_nvidia.so W0506 20:06:53.451930 416509 nvc_info.c:307] missing compat32 library libnvidia-cfg.so W0506 20:06:53.451943 416509 nvc_info.c:307] missing compat32 library libvdpau_nvidia.so W0506 20:06:53.451955 416509 nvc_info.c:307] missing compat32 library libnvidia-rtcore.so W0506 20:06:53.451967 416509 nvc_info.c:307] missing compat32 library libnvoptix.so W0506 20:06:53.451975 416509 nvc_info.c:307] missing compat32 library libnvidia-cbl.so I0506 20:06:53.452328 416509 nvc_info.c:233] selecting /usr/bin/nvidia-smi I0506 20:06:53.452362 416509 nvc_info.c:233] selecting /usr/bin/nvidia-debugdump I0506 20:06:53.452393 416509 nvc_info.c:233] selecting /usr/bin/nvidia-persistenced I0506 20:06:53.452423 416509 nvc_info.c:233] selecting /usr/bin/nvidia-cuda-mps-control I0506 20:06:53.452453 416509 nvc_info.c:233] selecting /usr/bin/nvidia-cuda-mps-server I0506 20:06:53.452494 416509 nvc_info.c:370] listing device /dev/nvidiactl I0506 20:06:53.452503 416509 nvc_info.c:370] listing device /dev/nvidia-uvm I0506 20:06:53.452517 416509 nvc_info.c:370] listing device /dev/nvidia-uvm-tools I0506 20:06:53.452525 416509 nvc_info.c:370] listing device /dev/nvidia-modeset I0506 20:06:53.452567 416509 nvc_info.c:274] listing ipc /run/nvidia-persistenced/socket W0506 20:06:53.452593 416509 nvc_info.c:278] missing ipc /tmp/nvidia-mps I0506 20:06:53.452602 416509 nvc_info.c:494] requesting device information with '' I0506 20:06:53.458383 416509 nvc_info.c:524] listing device /dev/nvidia0 (GPU-7ac1e178-9059-c612-045c-4534601f45b0 at 00000000:1d:00.0) NVRM version: 440.64 CUDA version: 10.2

Device Index: 0 Device Minor: 0 Model: GeForce GTX 1060 3GB Brand: GeForce GPU UUID: GPU-7ac1e178-9059-c612-045c-4534601f45b0 Bus Location: 00000000:1d:00.0 Architecture: 6.1 I0506 20:06:53.458442 416509 nvc.c:318] shutting down library context I0506 20:06:53.459080 416511 driver.c:192] terminating driver service I0506 20:06:53.575353 416509 driver.c:233] driver service terminated successfully `

[x] Kernel version from uname -a Linux ubuntu 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[x] Driver information from nvidia-smi -a `==============NVSMI LOG==============

Timestamp : Wed May 6 16:11:11 2020 Driver Version : 440.64 CUDA Version : 10.2

Attached GPUs : 1 GPU 00000000:1D:00.0 Product Name : GeForce GTX 1060 3GB Product Brand : GeForce Display Mode : Enabled Display Active : Enabled Persistence Mode : Disabled Accounting Mode : Disabled Accounting Mode Buffer Size : 4000 Driver Model Current : N/A Pending : N/A Serial Number : N/A GPU UUID : GPU-7ac1e178-9059-c612-045c-4534601f45b0 Minor Number : 0 VBIOS Version : 86.06.59.00.60 MultiGPU Board : No Board ID : 0x1d00 GPU Part Number : N/A Inforom Version Image Version : G001.0000.01.04 OEM Object : 1.1 ECC Object : N/A Power Management Object : N/A GPU Operation Mode Current : N/A Pending : N/A GPU Virtualization Mode Virtualization Mode : None Host VGPU Mode : N/A IBMNPU Relaxed Ordering Mode : N/A PCI Bus : 0x1D Device : 0x00 Domain : 0x0000 Device Id : 0x1C0210DE Bus Id : 00000000:1D:00.0 Sub System Id : 0x61623842 GPU Link Info PCIe Generation Max : 3 Current : 1 Link Width Max : 16x Current : 16x Bridge Chip Type : N/A Firmware : N/A Replays Since Reset : 0 Replay Number Rollovers : 0 Tx Throughput : 10000 KB/s Rx Throughput : 294000 KB/s Fan Speed : 5 % Performance State : P8 Clocks Throttle Reasons Idle : Active Applications Clocks Setting : Not Active SW Power Cap : Not Active HW Slowdown : Not Active HW Thermal Slowdown : Not Active HW Power Brake Slowdown : Not Active Sync Boost : Not Active SW Thermal Slowdown : Not Active Display Clock Setting : Not Active FB Memory Usage Total : 3011 MiB Used : 253 MiB Free : 2758 MiB BAR1 Memory Usage Total : 256 MiB Used : 7 MiB Free : 249 MiB Compute Mode : Default Utilization Gpu : 0 % Memory : 6 % Encoder : 0 % Decoder : 0 % Encoder Stats Active Sessions : 0 Average FPS : 0 Average Latency : 0 FBC Stats Active Sessions : 0 Average FPS : 0 Average Latency : 0 Ecc Mode Current : N/A Pending : N/A ECC Errors Volatile Single Bit
Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Texture Memory : N/A Texture Shared : N/A CBU : N/A Total : N/A Double Bit
Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Texture Memory : N/A Texture Shared : N/A CBU : N/A Total : N/A Aggregate Single Bit
Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Texture Memory : N/A Texture Shared : N/A CBU : N/A Total : N/A Double Bit
Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Texture Memory : N/A Texture Shared : N/A CBU : N/A Total : N/A Retired Pages Single Bit ECC : N/A Double Bit ECC : N/A Pending Page Blacklist : N/A Temperature GPU Current Temp : 50 C GPU Shutdown Temp : 102 C GPU Slowdown Temp : 99 C GPU Max Operating Temp : N/A Memory Current Temp : N/A Memory Max Operating Temp : N/A Power Readings Power Management : Supported Power Draw : 9.29 W Power Limit : 120.00 W Default Power Limit : 120.00 W Enforced Power Limit : 120.00 W Min Power Limit : 60.00 W Max Power Limit : 140.00 W Clocks Graphics : 139 MHz SM : 139 MHz Memory : 405 MHz Video : 544 MHz Applications Clocks Graphics : N/A Memory : N/A Default Applications Clocks Graphics : N/A Memory : N/A Max Clocks Graphics : 2012 MHz SM : 2012 MHz Memory : 4004 MHz Video : 1708 MHz Max Customer Boost Clocks Graphics : N/A Clock Policy Auto Boost : N/A Auto Boost Default : N/A Processes Process ID : 1547 Type : G Name : /usr/lib/xorg/Xorg Used GPU Memory : 114 MiB Process ID : 15526 Type : G Name : /usr/bin/kwin_x11 Used GPU Memory : 52 MiB Process ID : 15538 Type : G Name : /usr/bin/plasmashell Used GPU Memory : 67 MiB Process ID : 16007 Type : G Name : /usr/bin/akonadi_archivemail_agent Used GPU Memory : 2 MiB Process ID : 16026 Type : G Name : /usr/bin/akonadi_mailfilter_agent Used GPU Memory : 2 MiB Process ID : 16034 Type : G Name : /usr/bin/akonadi_sendlater_agent Used GPU Memory : 2 MiB Process ID : 16035 Type : G Name : /usr/bin/akonadi_unifiedmailbox_agent Used GPU Memory : 2 MiB Process ID : 391837 Type : G Name : /usr/lib/firefox/firefox Used GPU Memory : 2 MiB `

[x] Docker version from docker version `Client: Docker Engine - Community Version: 19.03.8 API version: 1.40 Go version: go1.12.17 Git commit: afacb8b7f0 Built: Wed Mar 11 01:25:46 2020 OS/Arch: linux/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 19.03.8 API version: 1.40 (minimum version 1.12) Go version: go1.12.17 Git commit: afacb8b7f0 Built: Wed Mar 11 01:24:19 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.13 GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429 runc: Version: 1.0.0-rc10 GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd docker-init: Version: 0.18.0 GitCommit: fec3683 `

[x] NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*' Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=============================-============-========================================================= un libgldispatch0-nvidia <none> <none> (no description available) ii libnvidia-cfg1-440:amd64 440.82+really.440.64-0ubuntu6 amd64 NVIDIA binary OpenGL/GLX configuration library un libnvidia-cfg1-any <none> <none> (no description available) un libnvidia-common <none> <none> (no description available) ii libnvidia-common-440 440.82+really.440.64-0ubuntu6 all Shared files used by the NVIDIA libraries rc libnvidia-compute-435:amd64 435.21-0ubuntu0.18.04.2 amd64 NVIDIA libcompute package ii libnvidia-compute-440:amd64 440.82+really.440.64-0ubuntu6 amd64 NVIDIA libcompute package ii libnvidia-compute-440:i386 440.82+really.440.64-0ubuntu6 i386 NVIDIA libcompute package ii libnvidia-container-tools 1.0.7-1 amd64 NVIDIA container runtime library (command-line tools) ii libnvidia-container1:amd64 1.0.7-1 amd64 NVIDIA container runtime library un libnvidia-decode <none> <none> (no description available) ii libnvidia-decode-440:amd64 440.82+really.440.64-0ubuntu6 amd64 NVIDIA Video Decoding runtime libraries ii libnvidia-decode-440:i386 440.82+really.440.64-0ubuntu6 i386 NVIDIA Video Decoding runtime libraries un libnvidia-encode <none> <none> (no description available) ii libnvidia-encode-440:amd64 440.82+really.440.64-0ubuntu6 amd64 NVENC Video Encoding runtime library ii libnvidia-encode-440:i386 440.82+really.440.64-0ubuntu6 i386 NVENC Video Encoding runtime library un libnvidia-extra <none> <none> (no description available) ii libnvidia-extra-440:amd64 440.82+really.440.64-0ubuntu6 amd64 Extra libraries for the NVIDIA driver un libnvidia-fbc1 <none> <none> (no description available) ii libnvidia-fbc1-440:amd64 440.82+really.440.64-0ubuntu6 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library ii libnvidia-fbc1-440:i386 440.82+really.440.64-0ubuntu6 i386 NVIDIA OpenGL-based Framebuffer Capture runtime library un libnvidia-gl <none> <none> (no description available) ii libnvidia-gl-440:amd64 440.82+really.440.64-0ubuntu6 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD ii libnvidia-gl-440:i386 440.82+really.440.64-0ubuntu6 i386 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD un libnvidia-ifr1 <none> <none> (no description available) ii libnvidia-ifr1-440:amd64 440.82+really.440.64-0ubuntu6 amd64 NVIDIA OpenGL-based Inband Frame Readback runtime library ii libnvidia-ifr1-440:i386 440.82+really.440.64-0ubuntu6 i386 NVIDIA OpenGL-based Inband Frame Readback runtime library un libnvidia-ml1 <none> <none> (no description available) un nvidia-304 <none> <none> (no description available) un nvidia-340 <none> <none> (no description available) un nvidia-384 <none> <none> (no description available) un nvidia-390 <none> <none> (no description available) un nvidia-common <none> <none> (no description available) rc nvidia-compute-utils-435 435.21-0ubuntu0.18.04.2 amd64 NVIDIA compute utilities ii nvidia-compute-utils-440 440.82+really.440.64-0ubuntu6 amd64 NVIDIA compute utilities un nvidia-container-runtime <none> <none> (no description available) un nvidia-container-runtime-hook <none> <none> (no description available) ii nvidia-container-toolkit 1.0.5-1 amd64 NVIDIA container runtime hook rc nvidia-dkms-435 435.21-0ubuntu0.18.04.2 amd64 NVIDIA DKMS package ii nvidia-dkms-440 440.82+really.440.64-0ubuntu6 amd64 NVIDIA DKMS package un nvidia-dkms-kernel <none> <none> (no description available) un nvidia-docker <none> <none> (no description available) rc nvidia-docker2 2.2.2-1 all nvidia-docker CLI wrapper ii nvidia-driver-440 440.82+really.440.64-0ubuntu6 amd64 NVIDIA driver metapackage un nvidia-driver-binary <none> <none> (no description available) un nvidia-kernel-common <none> <none> (no description available) rc nvidia-kernel-common-435 435.21-0ubuntu0.18.04.2 amd64 Shared files used with the kernel module ii nvidia-kernel-common-440 440.82+really.440.64-0ubuntu6 amd64 Shared files used with the kernel module un nvidia-kernel-source <none> <none> (no description available) un nvidia-kernel-source-435 <none> <none> (no description available) ii nvidia-kernel-source-440 440.82+really.440.64-0ubuntu6 amd64 NVIDIA kernel source package un nvidia-legacy-304xx-vdpau-driver <none> <none> (no description available) un nvidia-legacy-340xx-vdpau-driver <none> <none> (no description available) un nvidia-libopencl1-dev <none> <none> (no description available) un nvidia-opencl-icd <none> <none> (no description available) un nvidia-persistenced <none> <none> (no description available) ii nvidia-prime 0.8.14 all Tools to enable NVIDIA's Prime ii nvidia-settings 440.64-0ubuntu1 amd64 Tool for configuring the NVIDIA graphics driver un nvidia-settings-binary <none> <none> (no description available) un nvidia-smi <none> <none> (no description available) un nvidia-utils <none> <none> (no description available) ii nvidia-utils-440 440.82+really.440.64-0ubuntu6 amd64 NVIDIA driver support binaries un nvidia-vdpau-driver <none> <none> (no description available) ii xserver-xorg-video-nvidia-440 440.82+really.440.64-0ubuntu6 amd64 NVIDIA binary Xorg driver
[ ] NVIDIA container library version from nvidia-container-cli -V version: 1.0.7 build date: 2020-01-21T18:59+00:00 build revision: b71f87c04b8eca8a16bf60995506c35c937347d9 build compiler: x86_64-linux-gnu-gcc-7 7.4.0 build platform: x86_64 build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
[ ] NVIDIA container library logs (see troubleshooting)
[x] Docker command, image and tag used `services: plex: image: linuxserver/plex container_name: plex runtime: nvidia networks:
- t2_proxy volumes:
- /plexmeta/config:/config
- /data/media:/data/media
- /plextrans:/transcode ports:
- "32400:32400"
- "32400:32400/udp"
- "32469:32469"
- "32469:32469/udp"
- "5355:5353/udp"
- "1900:1900/udp" environment:
- TZ=$TZ
- PLEX_UID=$PUID
- PLEX_GID=$PGID
- VERSION=docker
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=all labels:
- "traefik.enable=true"
  HTTP Routers
- "traefik.http.routers.plexms-rtr.entrypoints=https"
- "traefik.http.routers.plexms-rtr.rule=Host(plex.$DOMAINNAME)"
- "traefik.http.routers.plexms-rtr.tls=true"
  Middlewares
- "traefik.http.routers.plexms-rtr.middlewares=chain-no-auth@file"
  HTTP Services
- "traefik.http.routers.plexms-rtr.service=plexms-svc"
- "traefik.http.services.plexms-svc.loadbalancer.server.port=32400"`

bigverm23 commented 4 years ago

runc: /usr/bin/runc docker-runc:

xfouloux commented 4 years ago

i have the exact same issue, container runs, then become unhealthy and sometime hangs forever. Can't stop or kill it.

# docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:25:55 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:24:26 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 nvidia:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

# nvidia-container-cli -k -d /dev/tty info

-- WARNING, the following logs are for debugging purposes only --

I0603 06:05:53.452519 30244 nvc.c:281] initializing library context (version=1.1.1, build=e5d6156aba457559979597c8e3d22c5d8d0622db)
I0603 06:05:53.452590 30244 nvc.c:255] using root /
I0603 06:05:53.452609 30244 nvc.c:256] using ldcache /etc/ld.so.cache
I0603 06:05:53.452614 30244 nvc.c:257] using unprivileged user 65534:65534
I0603 06:05:53.453423 30245 nvc.c:191] loading kernel module nvidia
I0603 06:05:53.453590 30245 nvc.c:203] loading kernel module nvidia_uvm
I0603 06:05:53.453726 30245 nvc.c:211] loading kernel module nvidia_modeset
I0603 06:05:53.453997 30246 driver.c:101] starting driver service

I0603 06:05:54.040620 30244 nvc_info.c:541] requesting driver information with ''
I0603 06:05:54.041759 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.440.64.00
I0603 06:05:54.041875 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvoptix.so.440.64.00
I0603 06:05:54.041914 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.440.64.00
I0603 06:05:54.041945 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.440.64.00
I0603 06:05:54.041980 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.440.64.00
I0603 06:05:54.042012 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.440.64.00
I0603 06:05:54.042060 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.440.64.00
I0603 06:05:54.042093 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.440.64.00
I0603 06:05:54.042128 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ifr.so.440.64.00
I0603 06:05:54.042176 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.440.64.00
I0603 06:05:54.042207 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.440.64.00
I0603 06:05:54.042239 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.440.64.00
I0603 06:05:54.042272 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.440.64.00
I0603 06:05:54.042321 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.440.64.00
I0603 06:05:54.042354 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.440.64.00
I0603 06:05:54.042401 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.440.64.00
I0603 06:05:54.042432 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.440.64.00
I0603 06:05:54.042465 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.440.64.00
I0603 06:05:54.042499 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cbl.so.440.64.00
I0603 06:05:54.042922 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.440.64.00
I0603 06:05:54.042993 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvcuvid.so.440.64.00
I0603 06:05:54.043134 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libcuda.so.440.64.00
I0603 06:05:54.043218 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.440.64.00
I0603 06:05:54.043254 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.440.64.00
I0603 06:05:54.043347 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.440.64.00
I0603 06:05:54.043399 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.440.64.00
I0603 06:05:54.043480 30244 nvc_info.c:155] selecting /usr/lib32/vdpau/libvdpau_nvidia.so.440.64.00
I0603 06:05:54.043517 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-tls.so.440.64.00
I0603 06:05:54.043549 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-ptxjitcompiler.so.440.64.00
I0603 06:05:54.043592 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-opticalflow.so.440.64.00
I0603 06:05:54.043674 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-opencl.so.440.64.00
I0603 06:05:54.043732 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-ml.so.440.64.00
I0603 06:05:54.044033 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-ifr.so.440.64.00
I0603 06:05:54.044082 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-glvkspirv.so.440.64.00
I0603 06:05:54.044240 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-glsi.so.440.64.00
I0603 06:05:54.044360 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-glcore.so.440.64.00
I0603 06:05:54.044396 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-fbc.so.440.64.00
I0603 06:05:54.044443 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-fatbinaryloader.so.440.64.00
I0603 06:05:54.044477 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-encode.so.440.64.00
I0603 06:05:54.044536 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-eglcore.so.440.64.00
I0603 06:05:54.044582 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-compiler.so.440.64.00
I0603 06:05:54.044965 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-allocator.so.440.64.00
I0603 06:05:54.045029 30244 nvc_info.c:155] selecting /usr/lib32/libnvcuvid.so.440.64.00
I0603 06:05:54.045092 30244 nvc_info.c:155] selecting /usr/lib32/libcuda.so.440.64.00
I0603 06:05:54.045156 30244 nvc_info.c:155] selecting /usr/lib32/libGLX_nvidia.so.440.64.00
I0603 06:05:54.045217 30244 nvc_info.c:155] selecting /usr/lib32/libGLESv2_nvidia.so.440.64.00
I0603 06:05:54.045252 30244 nvc_info.c:155] selecting /usr/lib32/libGLESv1_CM_nvidia.so.440.64.00
I0603 06:05:54.045332 30244 nvc_info.c:155] selecting /usr/lib32/libEGL_nvidia.so.440.64.00
W0603 06:05:54.045353 30244 nvc_info.c:310] missing compat32 library libnvidia-cfg.so
W0603 06:05:54.045374 30244 nvc_info.c:310] missing compat32 library libnvidia-rtcore.so
W0603 06:05:54.045381 30244 nvc_info.c:310] missing compat32 library libnvoptix.so
W0603 06:05:54.045388 30244 nvc_info.c:310] missing compat32 library libnvidia-cbl.so
I0603 06:05:54.045616 30244 nvc_info.c:236] selecting /usr/bin/nvidia-smi
I0603 06:05:54.045650 30244 nvc_info.c:236] selecting /usr/bin/nvidia-debugdump
I0603 06:05:54.045671 30244 nvc_info.c:236] selecting /usr/bin/nvidia-persistenced
I0603 06:05:54.045717 30244 nvc_info.c:236] selecting /usr/bin/nvidia-cuda-mps-control
I0603 06:05:54.045739 30244 nvc_info.c:236] selecting /usr/bin/nvidia-cuda-mps-server
I0603 06:05:54.045766 30244 nvc_info.c:373] listing device /dev/nvidiactl
I0603 06:05:54.045787 30244 nvc_info.c:373] listing device /dev/nvidia-uvm
I0603 06:05:54.045793 30244 nvc_info.c:373] listing device /dev/nvidia-uvm-tools
I0603 06:05:54.045813 30244 nvc_info.c:373] listing device /dev/nvidia-modeset
W0603 06:05:54.045838 30244 nvc_info.c:281] missing ipc /var/run/nvidia-persistenced/socket
W0603 06:05:54.045855 30244 nvc_info.c:281] missing ipc /tmp/nvidia-mps
I0603 06:05:54.045863 30244 nvc_info.c:598] requesting device information with ''
I0603 06:05:54.052179 30244 nvc_info.c:637] listing device /dev/nvidia0 (GPU-29e82e48-2d13-0906-4209-b923eaed182c at 00000000:01:00.0)
NVRM version:   440.64.00
CUDA version:   10.2

Device Index:   0
Device Minor:   0
Model:          GeForce RTX 2080
Brand:          GeForce
GPU UUID:       GPU-29e82e48-2d13-0906-4209-b923eaed182c
Bus Location:   00000000:01:00.0
Architecture:   7.5
I0603 06:05:54.052260 30244 nvc.c:318] shutting down library context
I0603 06:05:54.122623 30246 driver.c:156] terminating driver service
I0603 06:05:54.122960 30244 driver.c:196] driver service terminated successfully

# uname -a
Linux srv-us01 5.3.0-42-generic #34-Ubuntu SMP Fri Feb 28 05:49:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

# nvidia-smi -a

==============NVSMI LOG==============

Timestamp                           : Tue Jun  2 23:06:41 2020
Driver Version                      : 440.64.00
CUDA Version                        : 10.2

Attached GPUs                       : 1
GPU 00000000:01:00.0
    Product Name                    : GeForce RTX 2080
    Product Brand                   : GeForce
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 4000
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : N/A
    GPU UUID                        : GPU-29e82e48-2d13-0906-4209-b923eaed182c
    Minor Number                    : 0
    VBIOS Version                   : 90.04.23.00.6C
    MultiGPU Board                  : No
    Board ID                        : 0x100
    GPU Part Number                 : N/A
    Inforom Version
        Image Version               : G001.0000.02.04
        OEM Object                  : 1.1
        ECC Object                  : N/A
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization Mode         : None
        Host VGPU Mode              : N/A
    IBMNPU
        Relaxed Ordering Mode       : N/A
    PCI
        Bus                         : 0x01
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x1E8710DE
        Bus Id                      : 00000000:01:00.0
        Sub System Id               : 0x37A71458
        GPU Link Info
            PCIe Generation
                Max                 : 3
                Current             : 3
            Link Width
                Max                 : 16x
                Current             : 8x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays Since Reset         : 0
        Replay Number Rollovers     : 0
        Tx Throughput               : 0 KB/s
        Rx Throughput               : 0 KB/s
    Fan Speed                       : 29 %
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Not Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
            HW Thermal Slowdown     : Not Active
            HW Power Brake Slowdown : Not Active
        Sync Boost                  : Not Active
        SW Thermal Slowdown         : Not Active
        Display Clock Setting       : Not Active
    FB Memory Usage
        Total                       : 7982 MiB
        Used                        : 0 MiB
        Free                        : 7982 MiB
    BAR1 Memory Usage
        Total                       : 256 MiB
        Used                        : 2 MiB
        Free                        : 254 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Encoder Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    FBC Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    Ecc Mode
        Current                     : N/A
        Pending                     : N/A
    ECC Errors
        Volatile
            SRAM Correctable        : N/A
            SRAM Uncorrectable      : N/A
            DRAM Correctable        : N/A
            DRAM Uncorrectable      : N/A
        Aggregate
            SRAM Correctable        : N/A
            SRAM Uncorrectable      : N/A
            DRAM Correctable        : N/A
            DRAM Uncorrectable      : N/A
    Retired Pages
        Single Bit ECC              : N/A
        Double Bit ECC              : N/A
        Pending Page Blacklist      : N/A
    Temperature
        GPU Current Temp            : 43 C
        GPU Shutdown Temp           : 100 C
        GPU Slowdown Temp           : 97 C
        GPU Max Operating Temp      : 88 C
        Memory Current Temp         : N/A
        Memory Max Operating Temp   : N/A
    Power Readings
        Power Management            : Supported
        Power Draw                  : 28.92 W
        Power Limit                 : 245.00 W
        Default Power Limit         : 245.00 W
        Enforced Power Limit        : 245.00 W
        Min Power Limit             : 105.00 W
        Max Power Limit             : 300.00 W
    Clocks
        Graphics                    : 1515 MHz
        SM                          : 1515 MHz
        Memory                      : 7000 MHz
        Video                       : 1395 MHz
    Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Default Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Max Clocks
        Graphics                    : 2175 MHz
        SM                          : 2175 MHz
        Memory                      : 7000 MHz
        Video                       : 1950 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A
    Processes                       : None

# runc -v
runc version 1.0.0-rc10
commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
spec: 1.0.1-dev

xfouloux commented 4 years ago

errors example in syslog :

Jun  2 23:00:34 localhost dockerd[29000]: time="2020-06-02T23:00:34.848705509-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:00:39 localhost dockerd[29000]: time="2020-06-02T23:00:39.928513601-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:00:44 localhost dockerd[29000]: time="2020-06-02T23:00:44.979770503-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:00:50 localhost dockerd[29000]: time="2020-06-02T23:00:50.068667408-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:00:55 localhost dockerd[29000]: time="2020-06-02T23:00:55.166875094-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:00 localhost dockerd[29000]: time="2020-06-02T23:01:00.253540016-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:05 localhost dockerd[29000]: time="2020-06-02T23:01:05.310847728-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:10 localhost dockerd[29000]: time="2020-06-02T23:01:10.361814219-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:15 localhost dockerd[29000]: time="2020-06-02T23:01:15.421091744-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:20 localhost dockerd[29000]: time="2020-06-02T23:01:20.489495520-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:25 localhost dockerd[29000]: time="2020-06-02T23:01:25.564978060-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:30 localhost dockerd[29000]: time="2020-06-02T23:01:30.642824691-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:35 localhost dockerd[29000]: time="2020-06-02T23:01:35.728625577-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:40 localhost dockerd[29000]: time="2020-06-02T23:01:40.796021565-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:45 localhost dockerd[29000]: time="2020-06-02T23:01:45.855327028-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:50 localhost dockerd[29000]: time="2020-06-02T23:01:50.897109067-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:01:55 localhost dockerd[29000]: time="2020-06-02T23:01:55.935982392-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:02:00 localhost dockerd[29000]: time="2020-06-02T23:02:00.992472871-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:02:06 localhost dockerd[29000]: time="2020-06-02T23:02:06.048200991-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:02:11 localhost dockerd[29000]: time="2020-06-02T23:02:11.088129573-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:02:16 localhost dockerd[29000]: time="2020-06-02T23:02:16.148516214-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:02:21 localhost dockerd[29000]: time="2020-06-02T23:02:21.200053309-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:02:26 localhost dockerd[29000]: time="2020-06-02T23:02:26.272985101-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:02:31 localhost dockerd[29000]: time="2020-06-02T23:02:31.321134306-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"
Jun  2 23:02:36 localhost dockerd[29000]: time="2020-06-02T23:02:36.362726890-07:00" level=warning msg="Health check for container e209b5be6fde4c9851df9ee85f4dfe5a1b1524cf939c6212c86be751d3b2abc8 error: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"process_linux.go:101: executing setns process caused \\\"exit status 1\\\"\": unknown"

C84186 commented 3 years ago

Hi, I'm looking into running a docker-compose based application with GPU support (jellyfin), and stumbled across your issue.

Just FYI - I think some of the markdown for your log formats didn't quite work - also, when you have a lot of verbose logs, a nice way to present them under GFM is like this:

Example

```bash
nvidia-container-cli -k -d /dev/tty info

Output of nvidia-container-cli -k -d /dev/tty info

``` -- WARNING, the following logs are for debugging purposes only -- I0603 06:05:53.452519 30244 nvc.c:281] initializing library context (version=1.1.1, build=e5d6156aba457559979597c8e3d22c5d8d0622db) I0603 06:05:53.452590 30244 nvc.c:255] using root / I0603 06:05:53.452609 30244 nvc.c:256] using ldcache /etc/ld.so.cache ...etcetera (imagine I put another 150 lines) ```


Specifically, GFM supports the `<details>` & `<summary>` tags from `html`.

### Rendered

The above renders to:

```bash
nvidia-container-cli -k -d /dev/tty info

Output of nvidia-container-cli -k -d /dev/tty info

``` -- WARNING, the following logs are for debugging purposes only -- I0603 06:05:53.452519 30244 nvc.c:281] initializing library context (version=1.1.1, build=e5d6156aba457559979597c8e3d22c5d8d0622db) I0603 06:05:53.452590 30244 nvc.c:255] using root / I0603 06:05:53.452609 30244 nvc.c:256] using ldcache /etc/ld.so.cache -- WARNING, the following logs are for debugging purposes only -- I0603 06:05:53.452519 30244 nvc.c:281] initializing library context (version=1.1.1, build=e5d6156aba457559979597c8e3d22c5d8d0622db) I0603 06:05:53.452590 30244 nvc.c:255] using root / I0603 06:05:53.452609 30244 nvc.c:256] using ldcache /etc/ld.so.cache I0603 06:05:53.452614 30244 nvc.c:257] using unprivileged user 65534:65534 I0603 06:05:53.453423 30245 nvc.c:191] loading kernel module nvidia I0603 06:05:53.453590 30245 nvc.c:203] loading kernel module nvidia_uvm I0603 06:05:53.453726 30245 nvc.c:211] loading kernel module nvidia_modeset I0603 06:05:53.453997 30246 driver.c:101] starting driver service I0603 06:05:54.040620 30244 nvc_info.c:541] requesting driver information with '' I0603 06:05:54.041759 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.440.64.00 I0603 06:05:54.041875 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvoptix.so.440.64.00 I0603 06:05:54.041914 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.440.64.00 I0603 06:05:54.041945 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.440.64.00 I0603 06:05:54.041980 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.440.64.00 I0603 06:05:54.042012 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.440.64.00 I0603 06:05:54.042060 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.440.64.00 I0603 06:05:54.042093 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.440.64.00 I0603 06:05:54.042128 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ifr.so.440.64.00 I0603 06:05:54.042176 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.440.64.00 I0603 06:05:54.042207 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.440.64.00 I0603 06:05:54.042239 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.440.64.00 I0603 06:05:54.042272 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.440.64.00 I0603 06:05:54.042321 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.440.64.00 I0603 06:05:54.042354 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.440.64.00 I0603 06:05:54.042401 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.440.64.00 I0603 06:05:54.042432 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.440.64.00 I0603 06:05:54.042465 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.440.64.00 I0603 06:05:54.042499 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cbl.so.440.64.00 I0603 06:05:54.042922 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.440.64.00 I0603 06:05:54.042993 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libnvcuvid.so.440.64.00 I0603 06:05:54.043134 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libcuda.so.440.64.00 I0603 06:05:54.043218 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.440.64.00 I0603 06:05:54.043254 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.440.64.00 I0603 06:05:54.043347 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.440.64.00 I0603 06:05:54.043399 30244 nvc_info.c:155] selecting /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.440.64.00 I0603 06:05:54.043480 30244 nvc_info.c:155] selecting /usr/lib32/vdpau/libvdpau_nvidia.so.440.64.00 I0603 06:05:54.043517 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-tls.so.440.64.00 I0603 06:05:54.043549 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-ptxjitcompiler.so.440.64.00 I0603 06:05:54.043592 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-opticalflow.so.440.64.00 I0603 06:05:54.043674 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-opencl.so.440.64.00 I0603 06:05:54.043732 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-ml.so.440.64.00 I0603 06:05:54.044033 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-ifr.so.440.64.00 I0603 06:05:54.044082 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-glvkspirv.so.440.64.00 I0603 06:05:54.044240 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-glsi.so.440.64.00 I0603 06:05:54.044360 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-glcore.so.440.64.00 I0603 06:05:54.044396 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-fbc.so.440.64.00 I0603 06:05:54.044443 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-fatbinaryloader.so.440.64.00 I0603 06:05:54.044477 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-encode.so.440.64.00 I0603 06:05:54.044536 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-eglcore.so.440.64.00 I0603 06:05:54.044582 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-compiler.so.440.64.00 I0603 06:05:54.044965 30244 nvc_info.c:155] selecting /usr/lib32/libnvidia-allocator.so.440.64.00 I0603 06:05:54.045029 30244 nvc_info.c:155] selecting /usr/lib32/libnvcuvid.so.440.64.00 I0603 06:05:54.045092 30244 nvc_info.c:155] selecting /usr/lib32/libcuda.so.440.64.00 I0603 06:05:54.045156 30244 nvc_info.c:155] selecting /usr/lib32/libGLX_nvidia.so.440.64.00 I0603 06:05:54.045217 30244 nvc_info.c:155] selecting /usr/lib32/libGLESv2_nvidia.so.440.64.00 I0603 06:05:54.045252 30244 nvc_info.c:155] selecting /usr/lib32/libGLESv1_CM_nvidia.so.440.64.00 I0603 06:05:54.045332 30244 nvc_info.c:155] selecting /usr/lib32/libEGL_nvidia.so.440.64.00 W0603 06:05:54.045353 30244 nvc_info.c:310] missing compat32 library libnvidia-cfg.so W0603 06:05:54.045374 30244 nvc_info.c:310] missing compat32 library libnvidia-rtcore.so W0603 06:05:54.045381 30244 nvc_info.c:310] missing compat32 library libnvoptix.so W0603 06:05:54.045388 30244 nvc_info.c:310] missing compat32 library libnvidia-cbl.so I0603 06:05:54.045616 30244 nvc_info.c:236] selecting /usr/bin/nvidia-smi I0603 06:05:54.045650 30244 nvc_info.c:236] selecting /usr/bin/nvidia-debugdump I0603 06:05:54.045671 30244 nvc_info.c:236] selecting /usr/bin/nvidia-persistenced I0603 06:05:54.045717 30244 nvc_info.c:236] selecting /usr/bin/nvidia-cuda-mps-control I0603 06:05:54.045739 30244 nvc_info.c:236] selecting /usr/bin/nvidia-cuda-mps-server I0603 06:05:54.045766 30244 nvc_info.c:373] listing device /dev/nvidiactl I0603 06:05:54.045787 30244 nvc_info.c:373] listing device /dev/nvidia-uvm I0603 06:05:54.045793 30244 nvc_info.c:373] listing device /dev/nvidia-uvm-tools I0603 06:05:54.045813 30244 nvc_info.c:373] listing device /dev/nvidia-modeset W0603 06:05:54.045838 30244 nvc_info.c:281] missing ipc /var/run/nvidia-persistenced/socket W0603 06:05:54.045855 30244 nvc_info.c:281] missing ipc /tmp/nvidia-mps I0603 06:05:54.045863 30244 nvc_info.c:598] requesting device information with '' I0603 06:05:54.052179 30244 nvc_info.c:637] listing device /dev/nvidia0 (GPU-29e82e48-2d13-0906-4209-b923eaed182c at 00000000:01:00.0) NVRM version: 440.64.00 CUDA version: 10.2 Device Index: 0 Device Minor: 0 Model: GeForce RTX 2080 Brand: GeForce GPU UUID: GPU-29e82e48-2d13-0906-4209-b923eaed182c Bus Location: 00000000:01:00.0 Architecture: 7.5 I0603 06:05:54.052260 30244 nvc.c:318] shutting down library context I0603 06:05:54.122623 30246 driver.c:156] terminating driver service I0603 06:05:54.122960 30244 driver.c:196] driver service terminated successfully ```

NVIDIA / nvidia-docker