Advertise a recommended way to scrape cadvisor container metrics

[Chewie]

(Note: this issue also serves as a braindump and an explanation for future googlers)

Currently in the grafana ecosystem, there are many ways to scrape container metrics, with not so obvious tradeoffs between them:

• The agent can scrape directly from each node kubelet, by hitting the /metrics/cadvisor endpoint. This works best when the agent runs inside the cluster with a ServiceAccount properly granted, then making use of the node role in the kubernetes discovery.

• Alternatively, the requests can be proxied through the API server by hitting the api/v1/nodes/<MY_NODE>/proxy/metrics/cadvisor endpoint. This is what the prometheus helm chart did by default, but they're moving away from that in favor of option 1 (see https://github.com/prometheus-community/helm-charts/pull/3081). I've seen various discussions online regarding the pros and cons of this solution, but I'm not competent enough to weigh in on it (from what I understand, pro is being able to scrape from outside the cluster, con is heavier load on the api server)

• The prometheus operator and the agent operator both deploy a kubelet service as an entrypoint that can be scraped with a ServiceMonitor. It's unclear to me whether this service provides a value besides being able to be used with a ServiceMonitor.

• The grafana agent can launch its own cadvisor via an integration. This integration is not yet available as a component in flow mode but should eventually be there (see grafana/agent#3162). I'm not sure if this integration is tailored for deployments outside of kubernetes, or is intended to be a replacement/improvement of the cadvisor bundled within the kubelet.

• Long term, the landscape of kubernetes metrics may change with the introduction of PodAndContainerStatsFromCRI, but I think this option can be safely ignored for now.

Assuming a deployment in flow mode via the helm chart, which seems to be the future, what option should be recommended? I'd lean in favor of option 1, but I'm interested in gathering feedback by the subject matter experts :) If a consensus is reached, this information could be added to the docs as part of a tutorial.

Cheers!

[GrafanaWriter]

This is a great series of points, but I'm removing the type/docs label until we have the issue addressed by the Agent team to develop a recommendation.

[caleb-devops]

It may help to use the config from the k8s-montitoring Helm chart: https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/templates/agent_config/_cadvisor.river.txt

[ScOut3R]

Someone might find this useful. I run a DaemonSet to collect pod logs so I use this same instance to scrape metrics from node local pods as described in the example.

This inspired me to do the same for the cadvisor exposed container metrics.

discovery.kubernetes "nodes" {
  role = "node"
}

discovery.relabel "local_node_only" {
    targets = discovery.kubernetes.nodes.targets

    rule {
        source_labels = ["__meta_kubernetes_node_name"]
        action = "keep"
        regex = env("HOSTNAME")
    }

    rule {
        replacement = "https"
        target_label = "__scheme__"
    }

    rule {
        replacement = "metrics/cadvisor"
        target_label = "__metrics_path__"
    }
}

prometheus.scrape "local_cadvisor" {
    targets = discovery.relabel.local_node_only.output
    forward_to = [prometheus.remote_write.mimir.receiver]
    scrape_interval = "15s"

    authorization {
        credentials_file = "/run/secrets/kubernetes.io/serviceaccount/token"
        type = "Bearer"
    }

    tls_config {
        ca_file = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
        insecure_skip_verify = false
    }
}

[portswigger-tim]

Here is what works for me runnin in clustered mode to collect metrics and OTLP data:

      discovery.kubernetes "k8s_nodes" {
        role = "node"
      }

      discovery.relabel "kubelet" {
        targets = discovery.kubernetes.k8s_nodes.targets

        rule {
          replacement  = "kubernetes.default.svc:443"
          target_label = "__address__"
        }

        rule {
          regex         = "(.+)"
          source_labels = ["__meta_kubernetes_node_name"]
          replacement   = "/api/v1/nodes/$1/proxy/metrics"
          target_label  = "__metrics_path__"
        }
      }

      prometheus.scrape "kubelet" {
        job_name = "integrations/kubernetes/kubelet"
        targets  = discovery.relabel.kubelet.output
        scheme   = "https"

        tls_config {
          server_name          = "kubernetes"
          ca_file              = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
          insecure_skip_verify = false
        }

        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        forward_to        = [prometheus.remote_write.mimir.receiver]
      }

      discovery.relabel "cadvisor" {
        targets = discovery.kubernetes.k8s_nodes.targets

        rule {
          replacement  = "kubernetes.default.svc:443"
          target_label = "__address__"
        }

        rule {
          regex         = "(.+)"
          source_labels = ["__meta_kubernetes_node_name"]
          replacement   = "/api/v1/nodes/$1/proxy/metrics/cadvisor"
          target_label  = "__metrics_path__"
        }
      }

      prometheus.scrape "cadvisor" {
        job_name = "integrations/kubernetes/cadvisor"
        targets  = discovery.relabel.cadvisor.output
        scheme   = "https"

        tls_config {
          server_name          = "kubernetes"
          ca_file              = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
          insecure_skip_verify = false
        }

        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        forward_to        = [prometheus.remote_write.mimir.receiver]
      }

[snk-actian]

@portswigger-tim , @ScOut3R I have tried both of the solutions you provided and still I could not get some of the metrics (for example container_fs_usage_bytes) ? Were you able to get this metric with your configurtion?

[hkhairy]

@portswigger-tim brilliant :clap: worked with me

[hkhairy]

@snk-actian Yes, I tried @portswigger-tim 's solution and I can see this metric.

Bear in mind that this is a snippet of the configuration, and the not the whole configuration. There's a component that's not configured in the answer, which is the prometheus.remote_write. You'll need to set it up.

[snk-actian]

Hi @hkhairy thanks for the reply and confirmation.

Yes I am currently using Grafana Alloy and have full configuration working as expected. I have tried to add this to my current config and in debug screen, I could not see anything..

As it is working for others, I suspect it is something like a "PEBKAC". I will re-try with more attention..

[hkhairy]

Hi @snk-actian , you're most welcome

Ah you're using the newer Alloy. Apologies for not stating that earlier, but we're using the older Grafana Agent. We haven't migrated to Alloy yet. I'm not sure if this syntax is compatible with Grafana Alloy.

[snk-actian]

Hi @snk-actian , you're most welcome

Ah you're using the newer Alloy. Apologies for not stating that earlier, but we're using the older Grafana Agent. We haven't migrated to Alloy yet. I'm not sure if this syntax is compatible with Grafana Alloy.

Yeah they both use same river syntax and I migrated from Grafana Agent Flow to Alloy without any issues on configuration. ;-)

[hkhairy]

@snk-actian Okay Great! Hope you manage to solve this issue :crossed_fingers:

Have you checked that maybe it's a metric relabel config that's dropping the metric?

Also, while researching I found a way to look at the cadvisor metrics directly. Maybe you can check this way to see if this metric is being exported by cadvisor in the first place or not

1. Run kubectl proxy to tunnel the kubernetes proxy to localhost. It will usually be localhost:8001
2. Visit http://localhost:8001/api/v1/nodes to get a JSON response, detailing the names for your nodes. Copy the name of the node
   3. alternatively, you could run kubectl get nodes
3. Then, visit http://localhost:8001/api/v1/nodes/<node-name>/proxy/metrics/cadvisor

You'll see the text representation of the prometheus metrics that the agent is scraping. Just CTRL+F on it and search for the metric you want to make sure it's being exported by cadvisor.

[snk-actian]

Hey @hkhairy thanks for that last tip to proxy tunneling.. That was the proof that I did not do something wrong and that it does not provide the metric.. I got only below, which I normally get from cadvisor thru Grafana Agent / Alloy. :-(

# HELP container_fs_usage_bytes Number of bytes that are consumed by the container on this filesystem.
# TYPE container_fs_usage_bytes gauge
container_fs_usage_bytes{container="",device="/dev/nvme0n1p1",id="/",image="",name="",namespace="",pod=""} 8.32131072e+09 1723645343622
container_fs_usage_bytes{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1723645343622
container_fs_usage_bytes{container="",device="/run",id="/",image="",name="",namespace="",pod=""} 2.84672e+06 1723645343622
container_fs_usage_bytes{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1723645343622

No metrics for any named container :-(

[portswigger-tim]

Could it be related to the issues that began occurring with the removal of the Docker shim? https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-removal-affects-you/#some-filesystem-metrics-are-missing-and-the-metrics-format-is-different

[snk-actian]

Yes, that is the case indeed @portswigger-tim . I thought the starting point of this issue was this Docker shim removal and did not think of re-mentioning that :-)

So those alternative ways are not solutions for the main problem. Which I believe from that link, only way is to use external cadvisor daemonset or kubelet-summary.. I reviewed that doc before but did not want to get into that additional daemonset before trying all other possibilities

[hkhairy]

@snk-actian welp, it turns out I was getting the same metric container_fs_usage_bytes with no container or pod label. I thought the metric itself didn't show up for you, but now I understand that it's practically useless without the container and pod labels.

When I told you I saw the metric, I only checked if the metric existed, and didn't look at its labels :man_facepalming:

So, yeah, I guess it has to do with the shim after all :joy: