Setting Nvidia as "default runtime" in Docker

kevinthedang commented 3 months ago

Issue

When installing Nvidia Container Toolkit, the runtime is not set to nvidia when installing and configuring.

Steps to reproduce

Follow the guild to installing Nvidia Container Toolkit found in Resources.
Below will be a step by step on a Linux server or using wsl.

Installation with Apt

Step 1. Configure the production repository:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Step 2. Update the packages list from the repository:

sudo apt-get update

Step 3. Install the Nvidia Container Toolkit:

sudo apt-get install -y nvidia-container-toolkit

Configurating with Docker

Step 1.Configure the container runtime by using the nvidia-ctk command:

sudo nvidia-ctk runtime configure --runtime=docker

The nvidia-ctk command modifies the /etc/docker/daemon.json file on the host. The file is updated so that Docker can use the NVIDIA Container Runtime.

Step 2. Restart the Docker daemon:

sudo systemctl restart docker

Thoughts

Do we need Rootless Mode for this to work?? I don't believe so, but it might be...

Resources

Guide to installing Nvidia Container Toolkit
GitHub repository for Nvidia Container Toolkit

kevinthedang commented 3 months ago

I HAD IT WORKING NOW IT BROKE AGAIN. The following is proof it worked for a moment but I did not have it running in docker yet :(

I will keep looking into this to make sure it does not revert for some reason...

Edit:

Current situation when installing Nvidia Container Toolkit. It should say nvidia is default but here we are.

kevinthedang commented 3 months ago

As of now, following the Installation and Configuration seems to work!

This is part of a solution for the performance issue #7

IMPORTANT NOTE: It seems like running the containers on a local machine that uses wsl will be funky and not work as intended... I will continue to test this problem, or else the commands must be done in a Linux Distribution.

kevinthedang / discord-ollama