Arjun-D7
commented
3 years ago Hi Morristm
Did you resolved this issue? i am facing the same issue in my OpenShift environment.
please let us know if you know the fix for this
Open morristm opened 3 years ago
Arjun-D7
commented
3 years ago Hi Morristm
Did you resolved this issue? i am facing the same issue in my OpenShift environment.
please let us know if you know the fix for this
morristm
commented
3 years ago Hi Arjun, I wasn’t able to get the operator to work, so I decided to install the CUDA support on my workers manually. I’m fairly certain I started with this link https://developer.nvidia.com/cuda-toolkit https://developer.nvidia.com/cuda-toolkit. Once I had CUDA working and could run standalone containers on each worker that access the GPUs I took the next steps to deploy the OpenShift resources that allow containers to access and share the GPUs.
Here are the notes I took while doing this, can’t guarantee they’re perfect, but should help a little.
I had forgotten about this, and have tested the Nvidia Operator recently but considering you’re asking I suspect its still intended for RCOS workers.
Good Luck, Tom
On Aug 17, 2021, at 2:11 PM, Arjun-D7 @.***> wrote:
Hi Morristm
Did you resolved this issue? i am facing the same issue in my OpenShift environment.
please let us know if you know the fix for this
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/NVIDIA/k8s-device-plugin/issues/228#issuecomment-900522475, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGF4UPAIKVZJU2N4QEENREDT5KQ4PANCNFSM4WUIE3YA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.
github-actions[bot]
commented
6 months ago This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed.
1. Installed NVIDA GPU Operator from Operator Hub in ROS 4.6.12 cluster. After creating a cluster-policy the nvidia-container-toolset and nvidia-driver-daemonset daemonset deploy pods to nodes that have GPUs installed on them. nvidia-driver-daemonset pods all fail trying to use rhocp-4.6-for-rhel-8-x86_64-rpms. These nodes run RHEL 7 so I would expect this to fail. Is there any support for this process for RHEL 7 compute nodes? If not are the steps I can follow to install the drivers manually on the node, then enable OpenShift and crio to allow use of the gpu's by containers.
2.Install NVIDIA GPU Operator then create a ClusterPolicy
3. The following is one of the log files from an nvidia-driver-daemonset pod
========== NVIDIA Software Installer ==========
echo -e 'Starting installation of NVIDIA driver version 450.80.02 for Linux kernel version 3.10.0-1160.11.1.el7.x86_64\n' Starting installation of NVIDIA driver version 450.80.02 for Linux kernel version 3.10.0-1160.11.1.el7.x86_64
exec
flock -n 3
echo 3507487
trap 'echo '\''Caught signal'\''; exit 1' HUP INT QUIT PIPE TERM
trap _shutdown EXIT
_unload_driver
rmmod_args=()
local rmmod_args
local nvidia_deps=0
local nvidia_refs=0
local nvidia_uvm_refs=0
local nvidia_modeset_refs=0
echo 'Stopping NVIDIA persistence daemon...' Stopping NVIDIA persistence daemon...
'[' -f /var/run/nvidia-persistenced/nvidia-persistenced.pid ']' Unloading NVIDIA driver kernel modules...
echo 'Unloading NVIDIA driver kernel modules...'
'[' -f /sys/module/nvidia_modeset/refcnt ']'
'[' -f /sys/module/nvidia_uvm/refcnt ']'
'[' -f /sys/module/nvidia/refcnt ']'
'[' 0 -gt 0 ']'
'[' 0 -gt 0 ']'
'[' 0 -gt 0 ']'
'[' 0 -gt 0 ']'
return 0
_unmount_rootfs
echo 'Unmounting NVIDIA driver rootfs...' Unmounting NVIDIA driver rootfs...
findmnt -r -o TARGET
grep /run/nvidia/driver
_kernel_requires_package
local proc_mount_arg=
echo 'Checking NVIDIA driver packages...' Checking NVIDIA driver packages...
[[ ! -d /usr/src/nvidia-450.80.02/kernel ]]
cd /usr/src/nvidia-450.80.02/kernel
proc_mount_arg='--proc-mount-point /lib/modules/3.10.0-1160.11.1.el7.x86_64/proc' ++ ls -d -1 'precompiled/**'
return 0
_update_package_cache
'[' '' '!=' builtin ']'
echo 'Updating the package cache...' Updating the package cache...
yum -q makecache
_install_prerequisites ++ mktemp -d
local tmp_dir=/tmp/tmp.z7Jo1Bq05H
trap 'rm -rf /tmp/tmp.z7Jo1Bq05H' EXIT
cd /tmp/tmp.z7Jo1Bq05H
echo 'Installing elfutils...' Installing elfutils...
dnf install -q -y elfutils-libelf.x86_64 elfutils-libelf-devel.x86_64
rm -rf /lib/modules/3.10.0-1160.11.1.el7.x86_64
mkdir -p /lib/modules/3.10.0-1160.11.1.el7.x86_64/proc Enabling RHOCP and EUS RPM repos...
echo 'Enabling RHOCP and EUS RPM repos...'
dnf config-manager --set-enabled rhocp-4.6-for-rhel-8-x86_64-rpms Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode.
dnf makecache --releasever= Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. Red Hat OpenShift Container Platform 4.6 for RH 1.3 MB/s | 2.0 MB 00:01 Red Hat Enterprise Linux 8 for x86_64 - BaseOS 290 B/s | 354 B 00:01 Errors during downloading metadata for repository 'rhel-8-for-x86_64-baseos-rpms':
dnf config-manager --set-disabled rhocp-4.6-for-rhel-8-x86_64-rpms Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode.
dnf config-manager --set-enabled rhel-8-for-x86_64-baseos-eus-rpms Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode.
dnf makecache --releasever= Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. Red Hat Enterprise Linux 8 for x86_64 - BaseOS 220 B/s | 354 B 00:01 Errors during downloading metadata for repository 'rhel-8-for-x86_64-baseos-rpms':
dnf config-manager --set-disabled rhel-8-for-x86_64-baseos-eus-rpms Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode.
echo 'Installing Linux kernel headers...' Installing Linux kernel headers...
dnf -q -y --releasever= install kernel-headers-3.10.0-1160.11.1.el7.x86_64 kernel-devel-3.10.0-1160.11.1.el7.x86_64 Error: Failed to download metadata for repo 'rhel-8-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried ++ rm -rf /tmp/tmp.z7Jo1Bq05H
_shutdown
_unload_driver
rmmod_args=()
local rmmod_args
local nvidia_deps=0
local nvidia_refs=0
local nvidia_uvm_refs=0
local nvidia_modeset_refs=0 Stopping NVIDIA persistence daemon...
echo 'Stopping NVIDIA persistence daemon...'
'[' -f /var/run/nvidia-persistenced/nvidia-persistenced.pid ']' Unloading NVIDIA driver kernel modules...
echo 'Unloading NVIDIA driver kernel modules...'
'[' -f /sys/module/nvidia_modeset/refcnt ']'
'[' -f /sys/module/nvidia_uvm/refcnt ']'
'[' -f /sys/module/nvidia/refcnt ']'
'[' 0 -gt 0 ']'
'[' 0 -gt 0 ']'
'[' 0 -gt 0 ']'
'[' 0 -gt 0 ']'
return 0
_unmount_rootfs
echo 'Unmounting NVIDIA driver rootfs...' Unmounting NVIDIA driver rootfs...
findmnt -r -o TARGET
grep /run/nvidia/driver
rm -f /run/nvidia/nvidia-driver.pid /run/kernel/postinst.d/update-nvidia-driver
return 0