“Nvidia-smi not found in $PATH” error

NOTE - Cross-posting from docker-snap#184 here as it might be more related to the container toolkit. I have tried removing the Docker snap and installing the engine through apt/get as suggested in Docker's docs, and the result is the same.

I’m running into the following issue:

dani@ymir:~$ docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown.
[0.498s] 127  dani@ymir:~$

Here’re my specs

- OS version ```shell lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 24.04.1 LTS Release: 24.04 Codename: noble ``` - Kernel ```shell Linux ymir 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux ``` - NVIDIA driver/CUDA on the host ```shell Thu Sep 26 17:05:57 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.120 Driver Version: 550.120 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ […] ``` - Snap version ```shell snap version snap 2.63.1+24.04 snapd 2.63.1+24.04 series 16 ubuntu 24.04 kernel 6.8.0-45-generic ``` - Docker info ```shell snap info docker name: docker summary: Docker container runtime publisher: Canonical✓ store-url: https://snapcraft.io/docker contact: https://github.com/docker-snap/docker-snap/issues?q= license: (Apache-2.0 AND MIT AND GPL-2.0) description: | Build and run container images with Docker. **Usage** * This build can only access files in the home directory. So Dockerfiles and all other files used in commands like `docker build`, `docker save` and `docker load` need to be in $HOME. * You can change the configuration of this build by modifying the files in `/var/snap/docker/current/`. * Additional certificates used by the Docker daemon to authenticate with registries need to be added in `/var/snap/docker/current/etc/docker/certs.d` (instead of `/etc/docker/certs.d`). This directory can be accessed by other snaps using the `docker-registry-certificates` content interface. **Running Docker as normal user** By default, Docker is only accessible with root privileges (`sudo`). If you want to use docker as a regular user, you need to add your user to the `docker` group. sudo addgroup --system docker sudo adduser $USER docker newgrp docker sudo snap disable docker sudo snap enable docker **Warning:** if you add your user to the `docker` group, it will have similar power as the `root` user. For details on how this impacts security in your system, see https://docs.docker.com/engine/security/#docker-daemon-attack-surface **Authors** This snap is built by Canonical based on source code published by Docker, Inc. It is not endorsed or published by Docker, Inc. Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries. Docker, Inc. and other parties may also have trademark rights in other terms used herein. commands: - docker.compose - docker - docker.help services: docker.dockerd: simple, enabled, active docker.nvidia-container-toolkit: oneshot, enabled, inactive snap-id: sLCsFAO8PKM5Z0fAKNszUOX0YASjQfeZ tracking: latest/stable refresh-date: 22 days ago, at 04:47 UTC channels: latest/stable: 24.0.5 2024-09-03 (2932) 138MB - latest/candidate: 24.0.5 2024-09-03 (2932) 138MB - latest/beta: 27.2.0 2024-09-19 (2963) 146MB - latest/edge: 27.2.0 2024-09-20 (2969) 146MB - core18/stable: 20.10.17 2023-03-13 (2746) 146MB - core18/candidate: ↑ core18/beta: ↑ core18/edge: ↑ installed: 24.0.5 (2932) 138MB - ```

Do you have any sense what might be going on? Thanks!

NVIDIA / nvidia-container-toolkit

“Nvidia-smi not found in $PATH” error #718