search

NVIDIA / gpu-operator

NVIDIA GPU Operator creates, configures, and manages GPUs in Kubernetes
https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html
Apache License 2.0
1.86k stars 303 forks source link

[Feature] Support a custom e2e validation #1117

Open changhyuni opened 5 days ago

changhyuni commented 5 days ago

Issue or feature description

I sometimes elastically manage GPU nodes to save costs, creating and deleting them as needed. It would be helpful to have a validation feature that assigns a GPU to a real pod to ensure it functions correctly when a node is provisioned and validated.

Questions:

Is this possible? If enabled, after all the basic validations pass, the nvidia-operator could taint the node, deploy a pod (e.g., an application called e2e-check), and assign the GPU to perform a specific task, such as a basic inference or any workload utilizing the GPU. If the gpu operate good, the taint would be removed from the node.

Or, is there an existing method that currently supports this? I could implement it myself, but it would be good if a validation feature like this could be added to the nvidia-operator.

Hexoplon commented 3 days ago

The gpu-operator-validator component supports perforing cuda validation by running a workload, by setting the WITH_WORKLOAD env variable to "true". This might help with what you are looking for, the values is listed in the default helm chart values file