Error when collecting VPA recommendation metrics

[luizsfer]

Describe what happened: Error when collecting VPA (VerticalPodAutoscalers) recommendation metrics. The existing metric in KMS (kube-metrics-state) was not being sent to DataDog.

Describe what you expected: The data generated by the VPA, transmitted to the KMS should be collected as metrics by the DataDog agent.

Steps to reproduce the issue: Install the VPA chart and also the DataDog agent chart

Additional environment details (Operating System, Cloud provider, etc): Any environment.

[luizsfer]

In _kubernetes_state_core_config.yaml, needs add the verticalpodautoscalers into collectors config

{{- define "kubeStateMetricsCore-config" -}}
kubernetes_state_core.yaml.default: |-
{{- if .Values.datadog.kubeStateMetricsCore.useClusterCheckRunners }}
  cluster_check: true
{{- end }}
  init_config:
  instances:
    - collectors:
{{- if .Values.datadog.kubeStateMetricsCore.collectSecretMetrics }}
      - secrets
{{- end }}
      - nodes
      - pods
      - services
      - resourcequotas
      - replicationcontrollers
      - limitranges
      - persistentvolumeclaims
      - persistentvolumes
      - namespaces
      - endpoints
      - daemonsets
      - deployments
      - replicasets
      - statefulsets
      - cronjobs
      - jobs
      - horizontalpodautoscalers
      - poddisruptionbudgets
      - storageclasses
      - volumeattachments
      - verticalpodautoscalers
{{- if .Values.datadog.kubeStateMetricsCore.useClusterCheckRunners }}
      skip_leader_election: true
{{- end }}
      labels_as_tags:
{{ .Values.datadog.kubeStateMetricsCore.labelsAsTags | toYaml | indent 10 }}
{{- end -}}

I tried open a PR but don't have permission to push.

[clamoriniere]

Hi @luizsfer,

I think the issue is real, indeed the VPA collector is missing from check config.

I tried open a PR but don't have permission to push.

You will need to create a fork to open a PR. Else we can work on our side to provide a fix.

[JacksonDavenport]

Hey @luizsfer We're taking a look at it. One wrinkle is that VPA isn't always enabled by default in Kubernetes clusters. This could result in spammy logs like the following each check run:

E0421 23:32:43.221328 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.5/tools/cache/reflector.go:167: Failed to watch v1beta2.VerticalPodAutoscaler: failed to list v1beta2.VerticalPodAutoscaler: the server could not find the requested resource (get verticalpodautoscalers.autoscaling.k8s.io)

So this might get put on hold while we take a look at that. In the meantime you can enable this manually by updating your Helm configuration like in the below example to create a custom kubernetes_state_core.yaml file. This should get picked up by the Cluster Agent for the check, rather than the pre-existing kubernetes_state_core.yaml.default file.

datadog:
  #(...)
clusterAgent:
  confd:
      kubernetes_state_core.yaml: |-
        init_config:
        instances:
          - collectors:
              - secrets
              - nodes
              - pods
              - services
              - resourcequotas
              - replicationcontrollers
              - limitranges
              - persistentvolumeclaims
              - persistentvolumes
              - namespaces
              - endpoints
              - daemonsets
              - deployments
              - replicasets
              - statefulsets
              - cronjobs
              - jobs
              - horizontalpodautoscalers
              - poddisruptionbudgets
              - storageclasses
              - volumeattachments
              - verticalpodautoscalers

[luizsfer]

Hi @JacksonDavenport and @clamoriniere Thanks for the help!

I saw that a PR has already been opened for fixing the problem. Thanks again, next time I will open the PR properly.

Happy to contribute.