Open eselyavka opened 4 months ago
@eselyavka, Please make sure that the containerd uses NVIDIA runtime: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html.
@nvvfedorov may be my runtime is pretty old on those DLAMI images
[root@test-machine ~]# yum info nvidia-container-toolkit
Loaded plugins: dkms-build-requires, extras_suggestions, kernel-livepatch, langpacks, priorities, update-motd, versionlock
Installed Packages
Name : nvidia-container-toolkit
Arch : x86_64
Version : 1.13.5
Release : 1
Size : 2.3 M
Repo : installed
From repo : libnvidia-container
Summary : NVIDIA Container Toolkit
URL : https://github.com/NVIDIA/nvidia-container-toolkit
License : Apache-2.0
Description : Provides tools and utilities to enable GPU support in containers.
I do not see any options in nvidia-ctk
to configure runtime for containerd
[root@test-machine ~]# nvidia-ctk runtime configure --help
NAME:
NVIDIA Container Toolkit CLI runtime configure - Add a runtime to the specified container engine
USAGE:
NVIDIA Container Toolkit CLI runtime configure [command options] [arguments...]
OPTIONS:
[GR_GG_COF_AWS_Sandbox_TempPoCAccess]
--dry-run update the runtime configuration as required but don't write changes to disk (default: false)
--runtime value the target runtime engine. One of [crio, docker] (default: "docker")
--config value path to the config file for the target runtime
--nvidia-runtime-name value specify the name of the NVIDIA runtime that will be added (default: "nvidia")
--runtime-path value specify the path to the NVIDIA runtime executable (default: "nvidia-container-runtime")
--set-as-default set the specified runtime as the default runtime (default: false)
--help, -h show help (default: false)
As you can see --runtime
accepting only [crio, docker]
no containerd
option.
I guess i have to try to update runtime to the latest version.
@eselyavka, The DCGM exporter depends on the Nvidia container runtime; please try to update the runtime configuration.
What is the version?
3.3.5-3.4.1-ubi9
What happened?
We are running dcgm-exported inside the containerd and on
g5.48xlarge
dcgm-exporter is struggling to come up online with this errorWhat did you expect to happen?
dcgm-exporter running without any issues and we can see this in the log output, for example we have dcgm-exporter running successfully on
g5.2xlarge
ec2 nodeWhat is the GPU model?
What is the environment?
We are spinning up kubernetes cluster via kops utility and running dcgm-exporter as a daemon sets on GPU ec2 instances. Version of the containerd is
Version of the kubelet is
How did you deploy the dcgm-exporter and what is the configuration?
We are deploying dcgm-exporter as a helm chart via argocd.
How to reproduce the issue?
Try to run dcgm-exporter under the containerd on
g5.48xlarge
ec2 instance which is using OSS DLAMI ImageAnything else we need to know?
No response