yevgeny-shnaidman
commented
6 months ago Hi Kakor, Openshift KMM does not support OKD, so it is ok that you are seeing all those errors :). Using DTK_AUTO requires Images and ImageStreams object to be correctly configured and maintained by the cluster, and i think that OKD is not there. Is there any specific reason why you need to use OKD instead of Openshift cluster?
Thanks, Yevgeny
On Tue, May 7, 2024 at 5:08 PM KakorHolloway @.***> wrote:
Hello,
First of all thank you very much for this operator.
I'm trying to use this solution on my OKD cluster in version 4.14 to install an AMD RADEON RX 7900XT card as GPU to test AI models.
Unfortunately I have a problem creating the image on the internal registry. I have this problem on kmm operator pod :
E0507 13:44:31.120907 1 controller.go:324] kmm "msg"="Reconciler error" "error"="failed to handle build for kernel version 6.5.5-200.fc38.x86_64: could not synchronize the build: could not make Build template: could not get DTK image for kernel 6.5.5-200.fc38.x86_64: could not find kernel 6.5.5-200.fc38.x86_64 in kernel --> OS mapping" "Module"={"name":"dc-harbor","namespace":"openshift-amd-gpu"} "controller"="BuildSignReconciler" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "name"="dc-harbor" "namespace"="openshift-amd-gpu" "reconcileID"="ae042701-0f38-4dfe-802b-56795b3f5691"
If I remove DeviceConfig to add the DTK image value of my cluster version on the Module object (I remove the owner reference), and if I change DTK_AUTO in the configuration map to DTK, I can start the build .
Even with this option, I have to add a new variable named "KVERS" and replace it with the kernel version of the DTK image. (And apply this in the config map to replace all KERNEL_VERSION variables with KVERS). (6.5.5-200.fc38.x86_64 to 5.14.0-284.36.1.el9_2.x86_64)
You will find my custom files attached. cm_and_modude.txt https://github.com/yevgeny-shnaidman/amd-gpu-operator/files/15236644/cm_and_modude.txt
If all those change are mades, I can build the image and use it by recreating the DeviceConfig file but It will not work as intended because of those logs :
I0507 13:56:47.805280 1 main.go:100] kmm-worker "msg"="Starting worker" "git commit"="e1625b2" "version"="2.0.1" I0507 13:56:47.805292 1 main.go:101] kmm-worker "msg"="Reading pull secrets" "base dir"="/var/run/kmm/pull-secrets" I0507 13:56:47.805362 1 pullsecrets.go:39] kmm-worker "msg"="Reading file" "path"="/var/run/kmm/pull-secrets/amd-gpu-operator-kmm-module-loader-dockercfg-wjwn5/..2024_05_07_13_31_23.2974230544/.dockercfg" "type"="kubernetes.io/dockercfg" I0507 13:56:47.805388 1 pullsecrets.go:35] kmm-worker "msg"="Unhandled file name; ignoring" "path"="/var/run/kmm/pull-secrets/amd-gpu-operator-kmm-module-loader-dockercfg-wjwn5/..data" I0507 13:56:47.805395 1 pullsecrets.go:39] kmm-worker "msg"="Reading file" "path"="/var/run/kmm/pull-secrets/amd-gpu-operator-kmm-module-loader-dockercfg-wjwn5/.dockercfg" "type"="kubernetes.io/dockercfg" I0507 13:56:47.807301 1 mirror.go:43] kmm-worker "msg"="No configuration found for registry" "image name"="harbor.kakor.ovh/public/amddriver:v1.0" I0507 13:56:47.807314 1 worker.go:190] kmm-worker "msg"="Pulling image" "image name"="harbor.kakor.ovh/public/amddriver:v1.0" time="2024-05-07T13:57:12Z" level=warning msg="ignored xattrs in archive: underlying filesystem doesn't support them" errors="[operation not permitted]" time="2024-05-07T13:57:12Z" level=warning msg="ignored xattrs in archive: underlying filesystem doesn't support them" errors="[operation not permitted]" I0507 13:57:16.304671 1 worker.go:198] kmm-worker "msg"="Image pulled successfully" "dir"="/var/run/kmm/images/harbor.kakor.ovh/public/amddriver:v1.0/fs" "image name"="/public/amddriver:v1.0" I0507 13:57:16.304743 1 worker.go:49] kmm-worker "msg"="Unloading in-tree module" "name"="amdgpu" I0507 13:57:16.304831 1 modprobe.go:33] kmm-worker "msg"="Running modprobe" "command"="/usr/sbin/modprobe -rv amdgpu" I0507 13:57:16.306316 1 worker.go:59] kmm-worker "msg"="preparing firmware for loading" "host mount directory"="/var/lib/firmware" "image directory"="/var/run/kmm/images/harbor.kakor.ovh/public/amddriver:v1.0/fs/firmwareDir/updates" I0507 13:57:16.614772 1 modprobe.go:33] kmm-worker "msg"="Running modprobe" "command"="/usr/sbin/modprobe -vd /var/run/kmm/images//public/amddriver:v1.0/fs/opt amdgpu" I0507 13:57:16.615904 1 cmdlog.go:71] kmm-worker/modprobe/stderr "msg"="modprobe: FATAL: Module amdgpu not found in directory /var/run/kmm/images/harbor.kakor.ovh/public/amddriver:v1.0/fs/opt/lib/modules/6.5.5-200.fc38.x86_64" E0507 13:57:16.616009 1 cmdutils.go:11] kmm-worker "msg"="Fatal error" "error"="error while waiting on the command: exit status 1"
Thank you in advance :)
Regards
— Reply to this email directly, view it on GitHub https://github.com/yevgeny-shnaidman/amd-gpu-operator/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOPNM5LW4YDLLGCHNTFSHZTZBDN5HAVCNFSM6AAAAABHLCYKESVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI4DGNBWHEZTOOI . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Hello,
First of all thank you very much for this operator.
I'm trying to use this solution on my OKD cluster in version 4.14 to install an AMD RADEON RX 7900XT card as GPU to test AI models.
Unfortunately I have a problem creating the image on the internal registry. I have this problem on kmm operator pod :
E0507 13:44:31.120907 1 controller.go:324] kmm "msg"="Reconciler error" "error"="failed to handle build for kernel version 6.5.5-200.fc38.x86_64: could not synchronize the build: could not make Build template: could not get DTK image for kernel 6.5.5-200.fc38.x86_64: could not find kernel 6.5.5-200.fc38.x86_64 in kernel --> OS mapping" "Module"={"name":"dc-harbor","namespace":"openshift-amd-gpu"} "controller"="BuildSignReconciler" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "name"="dc-harbor" "namespace"="openshift-amd-gpu" "reconcileID"="ae042701-0f38-4dfe-802b-56795b3f5691"If I remove DeviceConfig to add the DTK image value of my cluster version on the Module object (I remove the owner reference), and if I change DTK_AUTO in the configuration map to DTK, I can start the build .
Even with this option, I have to add a new variable named "KVERS" and replace it with the kernel version of the DTK image. (And apply this in the config map to replace all KERNEL_VERSION variables with KVERS). (6.5.5-200.fc38.x86_64 to 5.14.0-284.36.1.el9_2.x86_64)
You will find my custom files attached. cm_and_modude.txt
If all those change are mades, I can build the image and use it by recreating the DeviceConfig file but It will not work as intended because of those logs :
Thank you in advance :)
Regards