[prometheus] prometheus-node-exporter and kube-state-metrics sub-charts are useless if prometheus annotations are not set

[zak905]

Describe the bug a clear and concise description of what the bug is.

The reason why a chart user would choose to enable prometheus-node-exporter and kube-state-metrics sub-charts is to collect their metrics. However, without the prometheus annotations prometheus.io/scrape and prometheus.io/port.

What's your helm version?

v3.14.4

What's your kubectl version?

v1.28.1

Which chart?

prometheus

What's the chart version?

25.21.0

What happened?

When installing the prometheus chart with kube-state-metrics.enabled=true and prometheus-node-exporter=true (these are the default values), one can notice that prometheus does not scrape metrics from those.

What you expected to happen?

I'd expect that prometheus starts scraping automatically from kube-state-metrics and prometheus-node-exporter

How to reproduce it?

installing the prometheus chart with kube-state-metrics.enabled=true and prometheus-node-exporter=true

Enter the changed values of values.yaml?

Adding pod annotations to both

kube-state-metrics:
  enabled: true
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
prometheus-node-exporter:
  enabled: true
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9100"

Enter the command that you execute and failing/misfunctioning.

helm install prometheus prometheus-community/prometheus

Anything else we need to know?

No response

[zeritti]

When installing the prometheus chart with kube-state-metrics.enabled=true and prometheus-node-exporter=true (these are the default values), one can notice that prometheus does not scrape metrics from those.

I cannot confirm the described results. Installing the chart with its default values shows Prometheus successfully discovering metrics endpoints of both kube-state-metrics and node exporter. As their services are annotated by the respective charts, Prometheus discovers both through scrape config kubernetes-service-endpoints (job label kubernetes-service-endpoints).

Further annotating the pods as suggested makes Prometheus discover them again through scrape config kubernetes-pods (job label kubernetes-pods).

[zak905]

Hi @zeritti, I just checked my config, and I found out that I have deleted the kubernetes-service-endpoints config. This is the reason why metrics were not scraped from kube-state-metrics and the node exporter. So the change is unnecessary, indeed. Thanks for checking