[kube-prometheus-stack] Kubernetes / Persistent Volumes Grafana dashboard empty

drewhemm commented 1 year ago

Describe the bug a clear and concise description of what the bug is.

This dashboard shows no data because the metric names referenced are not available in Prometheus:

kubelet_volume_stats_capacity_bytes and kubelet_volume_stats_available_bytes are not found, while kube_persistentvolume_capacity_bytes and kube_persistentvolumeclaim_resource_requests_storage_bytes are.

The metrics also have different labels, which means that swapping out the metric names alone will not work.

What's your helm version?

v3.11.3

What's your kubectl version?

v1.26.4

Which chart?

kube-prometheus-stack

What's the chart version?

48.2.0

What happened?

The Kubernetes / Persistent Volumes dashboard does not display any data:

What you expected to happen?

I expect to see data in the Kubernetes / Persistent Volumes dashboard.

How to reproduce it?

Deploy kube-prometheus-stack chart
Open the Kubernetes / Persistent Volumes dashboard in Grafana and you should see no data
Open the Prometheus Graph page
Search for metric names kubelet_volume_stats_capacity_bytes or kubelet_volume_stats_available_bytes; you will not find them

Enter the changed values of values.yaml?

grafana:
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - disableDeletion: false
        editable: true
        name: default
        options:
          path: /var/lib/grafana/dashboards/default
        orgId: 1
        type: file
  initChownData:
    enabled: false
  persistence:
    enabled: true
    storageClassName: efs
prometheus:
  prometheusSpec:
    podMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 50Gi
          storageClassName: efs

Enter the command that you execute and failing/misfunctioning.

helm install kube-prometheus-stack https://prometheus-community.github.io/helm-charts/kube-prometheus-stack --version 48.2.0 --values values.yaml

Anything else we need to know?

No response

meetzuber commented 1 year ago

I am also using same issue on kube-prometheus-stack-51.9.4, Any update/workaround.

deefdragon commented 1 year ago

This appears to be an issue with kubelet no longer returning some labels due to the removal of docker from k8s. See this comment in a similar issue

shusriva commented 3 months ago

Hi Folks, I am facing the same issue. My persistent volume dashboard data is empty. Can anyone suggest fix for this issue

prometheus-community / helm-charts