node-feature-discovery of gpu-operator sends excessive LIST requests to the API server

[jslouisyou]

1. Quick Debug Information

• OS/Version(e.g. RHEL8.6, Ubuntu22.04): Ubuntu 20.04.4 LTS
• Kernel Version: 5.4.0-113-generic
• Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): containerd://1.5.8
• K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): k8s v.1.21.6
• GPU Operator Version: v23.3.2

2. Issue or feature description

Hello, NVIDIA gpu-operator team.

I'm not sure if it's appropriate to post this issue here, since node-feature-discovery is maintained from kubernetes-sigs. If you think it's not suitable to post it here, please let me know.

Recently I got several alerts from K8S cluster which describes that API server tooks so long time to serve a LIST request from gpu-operator. Here's the alert and rule that I'm using:

• Alert:

  Long API server 99%-tile Latency
  LIST: 29.90 seconds while nfd.k8s-sigs.io/v1alpha1/nodefeatures request.

• Rule: histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{subresource!~"(log|exec|portforward|proxy)",verb!~"^(?:CONNECT|WATCHLIST|WATCH)$"} [10m])) WITHOUT (instance)) > 10

I also found all gpu-operator-node-feature-discovery-worker pods are tried to send GET verb to API server to query the nodefeatures resource (assumed that this pod needed to get information about node labels). Here's the part of audit log:

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"df926f36-8c1f-488e-ac88-11690e24660a","stage":"ResponseComplete","requestURI":"/apis/nfd.k8s-sigs.io/v1alpha1/namespaces/gpu-operator/nodefeatures/sra100-033","verb":"get","user":{"username":"system:serviceaccount:gpu-operator:node-feature-discovery","uid":"da2306ea-536f-455d-bf18-817299dd5489","groups":["system:serviceaccounts","system:serviceaccounts:gpu-operator","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["gpu-operator-node-feature-discovery-worker-49qq6"],"authentication.kubernetes.io/pod-uid":["65dfb997-221e-4a5c-92df-7ff111ea6137"]}},"sourceIPs":["75.17.103.53"],"userAgent":"nfd-worker/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"nodefeatures","namespace":"gpu-operator","name":"sra100-033","apiGroup":"nfd.k8s-sigs.io","apiVersion":"v1alpha1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2024-08-07T01:35:20.355504Z","stageTimestamp":"2024-08-07T01:35:20.676700Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"gpu-operator-node-feature-discovery\" of ClusterRole \"gpu-operator-node-feature-discovery\" to ServiceAccount \"node-feature-discovery/gpu-operator\""}}

I think this is strange that it takes this long to process LIST requests when my k8s cluster only has 300 GPU nodes and why node-feature-discovery-worker pods are sending GET request every minute.

Do you have any information about this problem? If there are any parameters that can be changed or if you could provide any ideas, I would be very grateful.

Thanks!

3. Steps to reproduce the issue

4. Information to attach (optional if deemed irrelevant)

• Logs from gpu-operator-node-feature-discovery-worker pods - query every minute

  I0727 13:17:09.124600       1 nfd-worker.go:459] starting feature discovery...
  I0727 13:17:09.125214       1 nfd-worker.go:471] feature discovery completed
  I0727 13:18:09.255614       1 local.go:115] starting hooks...
  I0727 13:18:09.446418       1 nfd-worker.go:459] starting feature discovery...
  I0727 13:18:09.446946       1 nfd-worker.go:471] feature discovery completed
  I0727 13:19:09.575466       1 local.go:115] starting hooks...
  I0727 13:19:09.858354       1 nfd-worker.go:459] starting feature discovery...
  I0727 13:19:09.858914       1 nfd-worker.go:471] feature discovery completed
  I0727 13:20:10.025155       1 local.go:115] starting hooks...
  .... and so on

[cdesiniotis]

@ArangoGutierrez

[jslouisyou]

Hello, I'm sorry but are there any updates from this issue?

[tariq1890]

Please raise an issue in https://github.com/kubernetes-sigs/node-feature-discovery