How to query the validation result using api?

[chenditc]

1. Quick Debug Information

• OS/Version(e.g. RHEL8.6, Ubuntu22.04): Mariner 2.0
• Kernel Version: 6.5.1
• Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): Containerd
• K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): K8s
• GPU Operator Version: v24.3.0

2. Issue or feature description

I can see the nvidia-operator-validator finishes its init container and completes validation by checking the code and the status. But the result seems not reported to anywhere, like node label.

I have a operator to deploy machine learning model, I want to check the gpu validation result before I deploy the model, so that I can ensure there is sufficient hardware instead of hanging on "FailedToSchedule" state or failed at runtime.

Is there a way to query the validation result? I am thinking using nodel label or some api like /metrics.

[cdesiniotis]

@chenditc one option is to enable the nvidia-node-status-exporter pod by enabling it in Clusterpolicy nodeStatusExporter.enabled=true. This component exposes the following metrics: https://github.com/NVIDIA/gpu-operator/blob/main/validator/metrics.go#L50-L68