klueska
commented
3 years ago Can you enable logging for the nvidia-container-cli in /etc/nvidia-container-runtime/cxonfig.toml and show me the output of it after (1) starting the first container, and (2) attempting the exec.
I.e.:
rm -rf /var /var/log/nvidia-container-toolkit.log
nvidia-docker run -d --name xxx -v /path/to/data:/container/path -it xxx:latest /bin/bash
docker exec -it container_name /bin/bash
cat /var /var/log/nvidia-container-toolkit.log
TomorrowIsAnOtherDay
I have start a container with command like
nvidia-docker run -d --name xxx -v /path/to/data:/container/path -it xxx:latest /bin/bashI successfully started the container from image at first time, and I can use it with GPU Then I try to enter the container with commanddocer exec -it container_name /bin/bashthe docker remind me thatnvidia-container-cli: mount error: namespace association failed: /proc/18331/ns/mnt: function not implemented
Then I stopped the container and try to start a new one with exactly same command above, but still have the error And the command
docker run -d --name xxx -v /path/to/data:/container/path -it xxx:latest /bin/bashcan start a container successfully but the GPU is not availableI searched all possible keywords try to solve the problem, but there is no one works
My version of nvidia-docker is 17.06.2-ce the version of nvidia-container-cli is version: 1.0.0 build date: 2018-01-11T00:23+0000 build revision: 4a618459e8ba522d834bb2b4c665847fae8ce0ad build compiler: gcc 4.8.5 20150623 (Red Hat 4.8.5-16) build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
The error while start a container with nvidia-docker is
docker: Error response from daemon: oci runtime error: container_linux.go:296: starting container process caused "process_linux.go:398: container init caused \"process_linux.go:381: running prestart hook 1 caused \\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig --device=all --compute --utility --require=cuda>=9.0 --pid=18331 /home/work/docker/devicemapper/mnt/9d23ec20f7616ee3d9b07bbdf8f411165f801cb062864530f4aa755064a950b2/rootfs]\\nnvidia-container-cli: mount error: namespace association failed: /proc/18331/ns/mnt: function not implemented\\n\\"\""
The output with command
nvidia-container-cli -k -d /dev/tty infois (only part of it): I1118 09:37:52.660599 77880 nvc_info.c:491] listing device /dev/nvidia7 (GPU-284781f8-551f-360d-bdcc-692edf6533f2 at 00000000:00:0f.0) NVRM version: 440.33.01 CUDA version: 10.2Device Index: 0 Model: Tesla V100-SXM2-32GB GPU UUID: GPU-ce2743d2-b6e3-e574-2e44-5ba47e7f0627 Bus Location: 00000000:00:08.0 Architecture: 7.0
Any advises or solution?