search

NVIDIA / nvidia-container-runtime

NVIDIA container runtime
Apache License 2.0
1.1k stars 159 forks source link

Cannot access GPU using Docker container nested in unprivileged LXD container #78

Closed klmmr closed 11 months ago

klmmr commented 4 years ago

Hi everyone, I am trying to use LXD with nested Docker for applications running on GPUs (e.g. deep learning with Tensorflow).

In my setup an unprivileged LXD container (usual Ubuntu 18.04 image from LXD) is running on a Ubuntu Server 18.04 Host. In this container nvidia-driver-430 (I can successfully execute nvidia-smi) and Docker (running hello-world container correctly) are installed. The GPU is passed from the host to the LXD container by mapping all devices in the LXD container (/dev/nvidia-uvm, /dev/nvidia-uvm-tools, /dev/nvidia0, /dev/nvidiactl).

In order to pass the GPU from the LXD container to a Docker container nvidia-container-runtime was installed within the LXD container and registered in /etc/docker/daemon.json.

The following error occurs when trying to execute a Docker container with GPU access:

$ docker run --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=0 nvidia/cuda:10.1-base nvidia-smi
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:413: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: write error: /sys/fs/cgroup/devices/docker/6e3c8640e6ff13268f1171c3d3ce4caa7326bb82d49bee22cfdf97fad91ced9b/devices.allow: operation not permitted\\\\n\\\"\"": unknown.
ERRO[0001] error waiting for container: context canceled

There seems to be some kind of permission problem writing to /sys/fs/cgroup/devices/docker/. Trying the same setup with Docker nested in a privileged LXD container (LXD option security.privileged true) seems to work fine. In that case no error occurs and the output from nvidia-smi (Docker container) is shown correctly.

However, privileged containers are not possible for my use case. Has anyone faced the same (or a similar) problem? Do you have an idea how to debug this problem further? I tried to debug using strace but didn't get any clue. The same applies for the logs (see below).

Please let me know when you have any questions or need some more information.

Background Information:

I am using a GeForce GTX 1080 Ti (using nvidia-driver-430) on Ubuntu Server 18.04.3, with the following versions of LXD and Docker.

LXD version (on host):

$ lxc version
Client version: 3.17
Server version: 3.17

Docker version (in LXD container)

$ docker version
Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:29:11 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:27:45 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Content of /var/log/nvidia-container-runtime.log for the error shown above:

2019/09/27 09:57:17 Running /usr/bin/nvidia-container-runtime
2019/09/27 09:57:17 Using bundle file: /run/containerd/io.containerd.runtime.v1.linux/moby/c4225c7f55683c1da8a1e9d6ed5feef5075b9d5e4323e45bdb870c198f6d5bd1/config.json
2019/09/27 09:57:17 prestart hook path: /usr/bin/nvidia-container-runtime-hook
2019/09/27 09:57:17 Prestart hook added, executing runc
2019/09/27 09:57:17 Looking for "docker-runc" binary
2019/09/27 09:57:17 "docker-runc" binary not found
2019/09/27 09:57:17 Looking for "runc" binary
2019/09/27 09:57:17 Runc path: /usr/bin/runc

Content of /var/log/nvidia-container-toolkit.log for the error shown above:

-- WARNING, the following logs are for debugging purposes only --

I0927 09:57:17.770959 718 nvc.c:281] initializing library context (version=1.0.5, build=13b836390888f7b7c7dca115d16d7e28ab15a836)
I0927 09:57:17.771114 718 nvc.c:255] using root /
I0927 09:57:17.771144 718 nvc.c:256] using ldcache /etc/ld.so.cache
I0927 09:57:17.771170 718 nvc.c:257] using unprivileged user 65534:65534
W0927 09:57:17.771261 718 nvc.c:166] skipping kernel modules load due to user namespace
I0927 09:57:17.771599 722 driver.c:133] starting driver service
I0927 09:57:18.072792 718 nvc_container.c:364] configuring container with 'compute utility supervised'
I0927 09:57:18.072947 718 nvc_container.c:212] selecting /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/local/cuda-10.1/compat/libcuda.so.418.87.00
I0927 09:57:18.072993 718 nvc_container.c:212] selecting /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/local/cuda-10.1/compat/libnvidia-fatbinaryloader.so.418.87.00
I0927 09:57:18.073019 718 nvc_container.c:212] selecting /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/local/cuda-10.1/compat/libnvidia-ptxjitcompiler.so.418.87.00
I0927 09:57:18.073189 718 nvc_container.c:384] setting pid to 693
I0927 09:57:18.073197 718 nvc_container.c:385] setting rootfs to /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged
I0927 09:57:18.073203 718 nvc_container.c:386] setting owner to 0:0
I0927 09:57:18.073209 718 nvc_container.c:387] setting bins directory to /usr/bin
I0927 09:57:18.073215 718 nvc_container.c:388] setting libs directory to /usr/lib/x86_64-linux-gnu
I0927 09:57:18.073220 718 nvc_container.c:389] setting libs32 directory to /usr/lib/i386-linux-gnu
I0927 09:57:18.073226 718 nvc_container.c:390] setting cudart directory to /usr/local/cuda
I0927 09:57:18.073232 718 nvc_container.c:391] setting ldconfig to @/sbin/ldconfig.real (host relative)
I0927 09:57:18.073238 718 nvc_container.c:392] setting mount namespace to /proc/693/ns/mnt
I0927 09:57:18.073243 718 nvc_container.c:394] setting devices cgroup to /sys/fs/cgroup/devices/docker/c4225c7f55683c1da8a1e9d6ed5feef5075b9d5e4323e45bdb870c198f6d5bd1
I0927 09:57:18.073252 718 nvc_info.c:437] requesting driver information with ''
I0927 09:57:18.073483 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvoptix.so.430.26
I0927 09:57:18.073530 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.430.26
I0927 09:57:18.073561 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.430.26
I0927 09:57:18.073594 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.430.26
I0927 09:57:18.073639 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.430.26
I0927 09:57:18.073684 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.430.26
I0927 09:57:18.073715 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.430.26
I0927 09:57:18.073761 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ifr.so.430.26
I0927 09:57:18.073805 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.430.26
I0927 09:57:18.073835 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.430.26
I0927 09:57:18.073866 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.430.26
I0927 09:57:18.073897 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.430.26
I0927 09:57:18.073942 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.430.26
I0927 09:57:18.073974 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.430.26
I0927 09:57:18.074016 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.430.26
I0927 09:57:18.074046 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.430.26
I0927 09:57:18.074079 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.430.26
I0927 09:57:18.074130 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvcuvid.so.430.26
I0927 09:57:18.074227 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libcuda.so.430.26
I0927 09:57:18.074295 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.430.26
I0927 09:57:18.074327 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.430.26
I0927 09:57:18.074359 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.430.26
I0927 09:57:18.074389 718 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.430.26
W0927 09:57:18.074408 718 nvc_info.c:302] missing library libvdpau_nvidia.so
W0927 09:57:18.074415 718 nvc_info.c:306] missing compat32 library libnvidia-ml.so
W0927 09:57:18.074421 718 nvc_info.c:306] missing compat32 library libnvidia-cfg.so
W0927 09:57:18.074427 718 nvc_info.c:306] missing compat32 library libcuda.so
W0927 09:57:18.074432 718 nvc_info.c:306] missing compat32 library libnvidia-opencl.so
W0927 09:57:18.074438 718 nvc_info.c:306] missing compat32 library libnvidia-ptxjitcompiler.so
W0927 09:57:18.074444 718 nvc_info.c:306] missing compat32 library libnvidia-fatbinaryloader.so
W0927 09:57:18.074450 718 nvc_info.c:306] missing compat32 library libnvidia-compiler.so
W0927 09:57:18.074455 718 nvc_info.c:306] missing compat32 library libvdpau_nvidia.so
W0927 09:57:18.074461 718 nvc_info.c:306] missing compat32 library libnvidia-encode.so
W0927 09:57:18.074467 718 nvc_info.c:306] missing compat32 library libnvidia-opticalflow.so
W0927 09:57:18.074473 718 nvc_info.c:306] missing compat32 library libnvcuvid.so
W0927 09:57:18.074478 718 nvc_info.c:306] missing compat32 library libnvidia-eglcore.so
W0927 09:57:18.074484 718 nvc_info.c:306] missing compat32 library libnvidia-glcore.so
W0927 09:57:18.074523 718 nvc_info.c:306] missing compat32 library libnvidia-tls.so
W0927 09:57:18.074529 718 nvc_info.c:306] missing compat32 library libnvidia-glsi.so
W0927 09:57:18.074535 718 nvc_info.c:306] missing compat32 library libnvidia-fbc.so
W0927 09:57:18.074541 718 nvc_info.c:306] missing compat32 library libnvidia-ifr.so
W0927 09:57:18.074546 718 nvc_info.c:306] missing compat32 library libnvidia-rtcore.so
W0927 09:57:18.074552 718 nvc_info.c:306] missing compat32 library libnvoptix.so
W0927 09:57:18.074558 718 nvc_info.c:306] missing compat32 library libGLX_nvidia.so
W0927 09:57:18.074564 718 nvc_info.c:306] missing compat32 library libEGL_nvidia.so
W0927 09:57:18.074569 718 nvc_info.c:306] missing compat32 library libGLESv2_nvidia.so
W0927 09:57:18.074575 718 nvc_info.c:306] missing compat32 library libGLESv1_CM_nvidia.so
W0927 09:57:18.074581 718 nvc_info.c:306] missing compat32 library libnvidia-glvkspirv.so
I0927 09:57:18.074825 718 nvc_info.c:232] selecting /usr/bin/nvidia-smi
I0927 09:57:18.074843 718 nvc_info.c:232] selecting /usr/bin/nvidia-debugdump
I0927 09:57:18.074861 718 nvc_info.c:232] selecting /usr/bin/nvidia-persistenced
I0927 09:57:18.074878 718 nvc_info.c:232] selecting /usr/bin/nvidia-cuda-mps-control
I0927 09:57:18.074896 718 nvc_info.c:232] selecting /usr/bin/nvidia-cuda-mps-server
I0927 09:57:18.074919 718 nvc_info.c:369] listing device /dev/nvidiactl
I0927 09:57:18.074925 718 nvc_info.c:369] listing device /dev/nvidia-uvm
I0927 09:57:18.074931 718 nvc_info.c:369] listing device /dev/nvidia-uvm-tools
I0927 09:57:18.074937 718 nvc_info.c:369] listing device /dev/nvidia-modeset
W0927 09:57:18.074958 718 nvc_info.c:277] missing ipc /var/run/nvidia-persistenced/socket
W0927 09:57:18.074972 718 nvc_info.c:277] missing ipc /tmp/nvidia-mps
I0927 09:57:18.074978 718 nvc_info.c:493] requesting device information with ''
I0927 09:57:18.081113 718 nvc_info.c:523] listing device /dev/nvidia0 (GPU-1b952bb0-b515-e51c-146a-b6625a6debc3 at 00000000:01:00.0)
I0927 09:57:18.081169 718 nvc_mount.c:252] mounting tmpfs at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/proc/driver/nvidia
I0927 09:57:18.081496 718 nvc_mount.c:73] mounting /usr/bin/nvidia-smi at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/bin/nvidia-smi
I0927 09:57:18.081545 718 nvc_mount.c:73] mounting /usr/bin/nvidia-debugdump at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/bin/nvidia-debugdump
I0927 09:57:18.081588 718 nvc_mount.c:73] mounting /usr/bin/nvidia-persistenced at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/bin/nvidia-persistenced
I0927 09:57:18.081630 718 nvc_mount.c:73] mounting /usr/bin/nvidia-cuda-mps-control at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/bin/nvidia-cuda-mps-control
I0927 09:57:18.081672 718 nvc_mount.c:73] mounting /usr/bin/nvidia-cuda-mps-server at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/bin/nvidia-cuda-mps-server
I0927 09:57:18.081790 718 nvc_mount.c:73] mounting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.430.26 at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.430.26
I0927 09:57:18.081836 718 nvc_mount.c:73] mounting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.430.26 at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.430.26
I0927 09:57:18.081878 718 nvc_mount.c:73] mounting /usr/lib/x86_64-linux-gnu/libcuda.so.430.26 at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/lib/x86_64-linux-gnu/libcuda.so.430.26
I0927 09:57:18.081924 718 nvc_mount.c:73] mounting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.430.26 at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.430.26
I0927 09:57:18.081968 718 nvc_mount.c:73] mounting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.430.26 at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.430.26
I0927 09:57:18.082017 718 nvc_mount.c:73] mounting /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.430.26 at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.430.26
I0927 09:57:18.082061 718 nvc_mount.c:73] mounting /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.430.26 at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.430.26
I0927 09:57:18.082082 718 nvc_mount.c:382] creating symlink /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/lib/x86_64-linux-gnu/libcuda.so -> libcuda.so.1
I0927 09:57:18.082168 718 nvc_mount.c:73] mounting /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/local/cuda-10.1/compat/libcuda.so.418.87.00 at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/lib/x86_64-linux-gnu/libcuda.so.418.87.00
I0927 09:57:18.082218 718 nvc_mount.c:73] mounting /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/local/cuda-10.1/compat/libnvidia-fatbinaryloader.so.418.87.00 at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.418.87.00
I0927 09:57:18.082265 718 nvc_mount.c:73] mounting /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/local/cuda-10.1/compat/libnvidia-ptxjitcompiler.so.418.87.00 at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.418.87.00
I0927 09:57:18.082312 718 nvc_mount.c:116] mounting /dev/nvidiactl at /var/lib/docker/overlay2/8bc61461c60de2257266d87d2defca62112cdbd67c4b7abb64db94771eae3ca8/merged/dev/nvidiactl
I0927 09:57:18.082336 718 nvc_mount.c:357] whitelisting device node 195:255
I0927 09:57:18.082829 718 nvc.c:318] shutting down library context
I0927 09:57:18.083007 722 driver.c:192] terminating driver service
I0927 09:57:18.182464 718 driver.c:233] driver service terminated successfully
davxiao commented 4 years ago

I am having the same errors when run docker run --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all --rm nvidia/cuda nvidia-smi It complains the following

nvidia-container-cli: mount error: write error: /sys/fs/cgroup/devices/docker/7f736858f0eb8fec8cce9b2a7dffc7646a58d65730482c25639cafa746350732/devices.allow: operation not permitted\\\\n\\\"\"": unknown.

It is in an unprivileged LXC container. Any thoughts? nvidia-smi runs just fine, nvidia-container-cli -k -d /dev/tty info produces the following output:

gpu 12|12:19 [~] nvidia-container-cli -k -d /dev/tty info

-- WARNING, the following logs are for debugging purposes only --

I1212 17:21:26.532332 1025 nvc.c:281] initializing library context (version=1.0.5, build=13b836390888f7b7c7dca115d16d7e28ab15a836)
I1212 17:21:26.532366 1025 nvc.c:255] using root /
I1212 17:21:26.532369 1025 nvc.c:256] using ldcache /etc/ld.so.cache
I1212 17:21:26.532372 1025 nvc.c:257] using unprivileged user 65534:65534
W1212 17:21:26.532392 1025 nvc.c:166] skipping kernel modules load due to user namespace
I1212 17:21:26.532504 1026 driver.c:133] starting driver service
I1212 17:21:26.995525 1025 nvc_info.c:437] requesting driver information with ''
I1212 17:21:26.995634 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.440.36
I1212 17:21:26.995671 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvoptix.so.440.36
I1212 17:21:26.995694 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.440.36
I1212 17:21:26.995711 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.440.36
I1212 17:21:26.995729 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.440.36
I1212 17:21:26.995746 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.440.36
I1212 17:21:26.995761 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.440.36
I1212 17:21:26.995778 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.440.36
I1212 17:21:26.995794 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ifr.so.440.36
I1212 17:21:26.995810 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.440.36
I1212 17:21:26.995826 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.440.36
I1212 17:21:26.995843 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.440.36
I1212 17:21:26.995860 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.440.36
I1212 17:21:26.995874 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.440.36
I1212 17:21:26.995890 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.440.36
I1212 17:21:26.995905 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.440.36
I1212 17:21:26.995921 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.440.36
I1212 17:21:26.995937 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.440.36
I1212 17:21:26.995953 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libnvcuvid.so.440.36
I1212 17:21:26.995994 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libcuda.so.440.36
I1212 17:21:26.996021 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.440.36
I1212 17:21:26.996037 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.440.36
I1212 17:21:26.996053 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.440.36
I1212 17:21:26.996069 1025 nvc_info.c:151] selecting /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.440.36
W1212 17:21:26.996077 1025 nvc_info.c:306] missing compat32 library libnvidia-ml.so
W1212 17:21:26.996079 1025 nvc_info.c:306] missing compat32 library libnvidia-cfg.so
W1212 17:21:26.996085 1025 nvc_info.c:306] missing compat32 library libcuda.so
W1212 17:21:26.996088 1025 nvc_info.c:306] missing compat32 library libnvidia-opencl.so
W1212 17:21:26.996091 1025 nvc_info.c:306] missing compat32 library libnvidia-ptxjitcompiler.so
W1212 17:21:26.996095 1025 nvc_info.c:306] missing compat32 library libnvidia-fatbinaryloader.so
W1212 17:21:26.996099 1025 nvc_info.c:306] missing compat32 library libnvidia-compiler.so
W1212 17:21:26.996102 1025 nvc_info.c:306] missing compat32 library libvdpau_nvidia.so
W1212 17:21:26.996104 1025 nvc_info.c:306] missing compat32 library libnvidia-encode.so
W1212 17:21:26.996107 1025 nvc_info.c:306] missing compat32 library libnvidia-opticalflow.so
W1212 17:21:26.996111 1025 nvc_info.c:306] missing compat32 library libnvcuvid.so
W1212 17:21:26.996114 1025 nvc_info.c:306] missing compat32 library libnvidia-eglcore.so
W1212 17:21:26.996117 1025 nvc_info.c:306] missing compat32 library libnvidia-glcore.so
W1212 17:21:26.996120 1025 nvc_info.c:306] missing compat32 library libnvidia-tls.so
W1212 17:21:26.996122 1025 nvc_info.c:306] missing compat32 library libnvidia-glsi.so
W1212 17:21:26.996125 1025 nvc_info.c:306] missing compat32 library libnvidia-fbc.so
W1212 17:21:26.996127 1025 nvc_info.c:306] missing compat32 library libnvidia-ifr.so
W1212 17:21:26.996130 1025 nvc_info.c:306] missing compat32 library libnvidia-rtcore.so
W1212 17:21:26.996133 1025 nvc_info.c:306] missing compat32 library libnvoptix.so
W1212 17:21:26.996135 1025 nvc_info.c:306] missing compat32 library libGLX_nvidia.so
W1212 17:21:26.996137 1025 nvc_info.c:306] missing compat32 library libEGL_nvidia.so
W1212 17:21:26.996140 1025 nvc_info.c:306] missing compat32 library libGLESv2_nvidia.so
W1212 17:21:26.996142 1025 nvc_info.c:306] missing compat32 library libGLESv1_CM_nvidia.so
W1212 17:21:26.996144 1025 nvc_info.c:306] missing compat32 library libnvidia-glvkspirv.so
I1212 17:21:26.996243 1025 nvc_info.c:232] selecting /usr/bin/nvidia-smi
I1212 17:21:26.996252 1025 nvc_info.c:232] selecting /usr/bin/nvidia-debugdump
I1212 17:21:26.996260 1025 nvc_info.c:232] selecting /usr/bin/nvidia-persistenced
I1212 17:21:26.996269 1025 nvc_info.c:232] selecting /usr/bin/nvidia-cuda-mps-control
I1212 17:21:26.996278 1025 nvc_info.c:232] selecting /usr/bin/nvidia-cuda-mps-server
I1212 17:21:26.996290 1025 nvc_info.c:369] listing device /dev/nvidiactl
I1212 17:21:26.996293 1025 nvc_info.c:369] listing device /dev/nvidia-uvm
I1212 17:21:26.996298 1025 nvc_info.c:369] listing device /dev/nvidia-uvm-tools
I1212 17:21:26.996302 1025 nvc_info.c:369] listing device /dev/nvidia-modeset
W1212 17:21:26.996317 1025 nvc_info.c:277] missing ipc /var/run/nvidia-persistenced/socket
W1212 17:21:26.996324 1025 nvc_info.c:277] missing ipc /tmp/nvidia-mps
I1212 17:21:26.996327 1025 nvc_info.c:493] requesting device information with ''
I1212 17:21:27.001816 1025 nvc_info.c:523] listing device /dev/nvidia0 (GPU-8d02206c-0145-a6d6-a681-fd178d12a183 at 00000000:09:00.0)
NVRM version:   440.36
CUDA version:   10.2

Device Index:   0
Device Minor:   0
Model:          GeForce GTX 1660 SUPER
Brand:          GeForce
GPU UUID:       GPU-8d02206c-0145-a6d6-a681-fd178d12a183
Bus Location:   00000000:09:00.0
Architecture:   7.5
I1212 17:21:27.001835 1025 nvc.c:318] shutting down library context
I1212 17:21:27.001974 1026 driver.c:192] terminating driver service
I1212 17:21:27.146646 1025 driver.c:233] driver service terminated successfully
playpusoffury commented 4 years ago

I also am experiencing this issue.

gvijqb commented 3 years ago

Any update on this?

cubercsl commented 3 years ago

Any update on this?

elezar commented 11 months ago

Given the changes in the architecture and the move to CDI, our LXC support would have to be revisited. If there is still a need, please open a new issue against https://github.com/NVIDIA/nvidia-container-toolkit with the requirements.