I see tempo resource dashboard using following promql queries.
sum by(pod) (rate(container_cpu_usage_seconds_total{cluster=~"$cluster", namespace=~"$namespace",container=~"ingester"}[$__rate_interval]))
The query is around how is the cluster label available as part of cadvisor metric. Containers metrics are scraped from cAdvisor which has pod and namespace labels. How does the pod label (For eg kubernetes_io_instance) get appended to cAdvisor metrics.
There needs to be some way to join data from other metric based on pod and namespace to get the label information
I referred following links. Both of them do not have any reference to how cluster (kubernetes_io_instance label) is available in the final metric that is written to remote write endpoint
I see there are other ways where you can join cadvisor metrics with kube_pod_labels based on pod name . But the existing resource dashboard queries expect cluster to be available which means there is some way to append the cluster label on to cAdvisor mertic
Can you share some pointers on how the kubernetes label value is appended to cadvisor metrics?
I see tempo resource dashboard using following promql queries.
sum by(pod) (rate(container_cpu_usage_seconds_total{cluster=~"$cluster", namespace=~"$namespace",container=~"ingester"}[$__rate_interval]))
The query is around how is the cluster label available as part of cadvisor metric. Containers metrics are scraped from cAdvisor which has pod and namespace labels. How does the pod label (For eg kubernetes_io_instance) get appended to cAdvisor metrics.
There needs to be some way to join data from other metric based on pod and namespace to get the label information
I referred following links. Both of them do not have any reference to how cluster (kubernetes_io_instance label) is available in the final metric that is written to remote write endpoint
https://grafana.com/docs/tempo/latest/operations/monitor/set-up-monitoring/ https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/otel-collector/
I see there are other ways where you can join cadvisor metrics with kube_pod_labels based on pod name . But the existing resource dashboard queries expect cluster to be available which means there is some way to append the cluster label on to cAdvisor mertic
Can you share some pointers on how the kubernetes label value is appended to cadvisor metrics?