Open lohwanjing opened 10 months ago
@lohwanjing can you run oc get imagesstream driver-toolkit -n openshift -o yaml
. Also logs from the operator pod will help.
Hi, attached is the imagestream logs. Note that we use ImageContentSourcePolicy to point quay.io to our internal mirror. Will get you the operator pod logs later
@shivamerla
Hi, apologies for the delay, attached are the logs for the operator pods, both before and after installing the CSP
gpu-operator-after-CSP.log gpu-operator-before-CSP.log
Additionally this is the list of pods in the namespace
@lohwanjing
Hi, did you manage to find any solution to the described problem ? I am also currently trying to install nvidia-operator in my OKD 4.15. I would appreciate any insight.
Cheers!
1. Quick Debug Information
2. Issue or feature description
When installing the GPU Operator in a fully air-gapped environment in OKD 4.13, I would expect the installation to complete successfully once the images and operators are mirrored accordingly with the appropriate ImageContentSourcePolicy set up, Currently, after the gpu-operator successfully installs, after setting up the default CSV, the daemonset installation fails with a crash-loop backoff. Based on the warning, the RHCOS image tag cannot be found, and it defaults back to entitlement fallback which fails due to being in a disconnected environment,
For OCP installations (and I would assume OKD as well), the gpu operator should be using the RHCOS container-toolkit, which would allow for entitlement-free installation of the NVIDIA Drivers, and therefore work in a disconnected environment.
3. Steps to reproduce the issue
4. Information to attach (optional if deemed irrelevant)
Note the tag on the daemonset which indicates the image is missing.
I saw issue #428, which was resolved by manually tagging the container toolkit container. In this case, manually tagging the container did not result in any changes.
Additionally, the container toolkit (at least from the docs) seems to run as a sidecar container, but not of the pods that were launched seem to contain the image within their specs.