Prometheus Unable to Scrape cAdvisor Metrics from Edge Nodes

[vofr]

What happened and what you expected to happen: I have deployed the Kube Prometheus stack on my KubeEdge cluster using Helm charts. However, I am facing an issue where Prometheus is unable to scrape cAdvisor metrics from the edge nodes because the kubelet on the edge nodes is exposing metrics over HTTP instead of HTTPS. This discrepancy in protocol is causing the scraping to fail.

Anything else we need to know?: To make the tailored kubelet expose port 10250 on Edge Nodes. I've made some changes in the edgecore.yaml set address 0.0.0.0 instead of localhost and readonlyport: 10250 from 10350.

Environment:

• Kubernetes version (use kubectl version): v1.28.12
• KubeEdge version(e.g. cloudcore --version and edgecore --version): cloudcore 1.17 edgecore 1.17

Is there a way to make the edge kubelet work with https?

[Piwriw]

have you try this ? https://kubeedge.io/docs/advanced/metrics/

[vofr]

have you try this ? https://kubeedge.io/docs/advanced/metrics/

I did, but Im not looking to deploy their metric server. Instead, I aim to expose the kubelet metrics to be scraped directly by Prometheus. To achieve this, I see two potential paths: A) Modify the customized Kubelet from EdgeCore to support HTTPS instead of HTTP. B) Configure Prometheus to scrape the Kubelet on port 10350 using HTTP for edge devices. I am seeking guidance on how to modify the Kubelet configuration for HTTPS, or if anyone can share their Prometheus setup that successfully scrapes metrics from KubeEdge, that would be greatly appreciated.

[Shelley-BaoYue]

Maybe you can ref to https://deploy-preview-578--competent-hypatia-4195b7.netlify.app/docs/advanced/prometheus

[vofr]

Maybe you can ref to https://deploy-preview-578--competent-hypatia-4195b7.netlify.app/docs/advanced/prometheus

The doc addresses the integration of Prometheus with Node Exporter, which collects host-level metrics and is functioning correctly. However, I need metrics specifically related to the pods running inside the host. Typically, these metrics are exposed by the Kubelet on port 10250 with HTTPS connection, but in the case of EdgeCore, the tailored Kubelet runs an HTTP server on port 10350. I am seeking a solution that either modifies the Kubelet to use HTTPS or a working Prometheus configuration example that successfully scrapes the tailored Kubelet over HTTP on edge devices.

[yeqiugt]

Maybe you can ref to https://deploy-preview-578--competent-hypatia-4195b7.netlify.app/docs/advanced/prometheus

If the ip of edgenode can't access, how to use prometheus gather metrics?

[JiaweiGithub]

We use fluent bit to collect and push it to Prometheus, which works.