[bug] difficult to understand what is missing

[shineeng]

Describe the bug

this would have be more appropriate as a "help request" since I don't think I'm dealing with a bug here.

My problem is that several panels in Kubernetes / Views / Pods are showing "no data".

Investigating this, and having read https://github.com/dotdc/grafana-dashboards-kubernetes?tab=readme-ov-file#known-issues I could not fully understand my issue.

For example, this is an example of panel's query:

sum(rate(container_cpu_usage_seconds_total{namespace="$namespace", pod=~"$pod", image!="", cluster="$cluster"}[$__rate_interval])) / sum(kube_pod_container_resource_requests{namespace="$namespace", pod=~"$pod", resource="cpu", job=~"$job", cluster="$cluster"})

this gives me "no data", but if I remove the condition image!="", then it starts working:

why do I have that image!="" condition? Should it be "filled" somehow?

How to reproduce?

I am using dashboards installed via helm chart providers, like described in https://github.com/dotdc/grafana-dashboards-kubernetes?tab=readme-ov-file#install-with-helm-values.

I am using grafana helm chart version 8.3.2

Expected behavior

I need to know how/when the panel's query SHOULD be filled, especially cluster="$cluster" and image!="". I can correcly see al my pods/namespaces in the respective combos/dropdown menu, but not "cluster":

Additional context

these dashboards are the best I could find for K8 environment.

great job

[shineeng]

My issue is probably related to missing "image" and "pod" info inside cadvisor:

[shineeng]

I think this is somewhat similar to https://github.com/prometheus-community/helm-charts/issues/3058, with reference to "originating" https://github.com/kubernetes/kubernetes/issues/111077

So if someone runs k8 over docker (or minikube) and uses this chart, will probably get this issue, as those labels are missing, some charts become infact broken.

[dotdc]

Are you using ArgoCD ? If yes, check if you have the same ArgoCD application name and Helm chart release name, this could be linked to https://github.com/prometheus-community/helm-charts/issues/1769#issuecomment-1170364587 ?

Otherwise, can you give more information on your setup? Kubernetes distribution & version, kube-prometheus-stack version, custom values...

[shineeng]

no, we are not using ArgoCD, it is infact a proof-of-concept project, a very small one actually.

Its "peculiarity" is that it started on docker-compose and was later moved to K8 with "konvert" tool, but it's still running on local docker desktop 4.32.0 over W11 in the end, mainly with experiment purposes. We're using Docker's "embedded" Kubernetes, which is at version v1.29.2 currently.

Over this local cluster, we're deploying prometheus chart v.25.23.0 in a separate namespace, along with grafana chart v.8.3.2 and other observability stack components (loki, tempo)

Everything else works pretty good, especially the node exporter dashboard or the more "global" ones. A couple dashboards are affected by these missing "image" and "pod" tho, making them sparse.

recap: Docker Desktop 4.32.0 on Win11 Enterprise 23H2 Kubernetes single-node cluster 1.29.2 prometheus chart v.25.23.0 grafana chart v.8.3.2 Helm version v3.15.2 imported your charts using helm and dashboardProviders approach, 80% working as expected.

[dotdc]

Unfortunately, I will not be able to replicate your setup on my side. Could you try with the kube-prometheus-stack chart instead ?

[shineeng]

well, it would take time to reconfigure all of my prometheus/tempo/loki DS, so I tried a "default" install just in order to check cadvisor tags, I now have the following:

as you can see, "pod" is now filled but "image" still isn't. I hope this helps, even tho I am installing with separate charts