Node labels for endpoint role doesn't seem to work on services without any runtime in the cluster

What did you do?

Using the prometheus-operator I'm trying to attach node metadata from a service that points to Kubelet.

The Kubelet Service:

# Manifest
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/managed-by: prometheus-operator
    app.kubernetes.io/name: kubelet
    k8s-app: kubelet
  name: kps-system-kubelet
  namespace: kube-system
spec:
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  - IPv6
  ipFamilyPolicy: RequireDualStack
  ports:
  - name: https-metrics
    port: 10250
    protocol: TCP
    targetPort: 10250
  - name: http-metrics
    port: 10255
    protocol: TCP
    targetPort: 10255
  - name: cadvisor
    port: 4194
    protocol: TCP
    targetPort: 4194
  sessionAffinity: None
  type: ClusterIP

# Describe
Name:              kps-system-kubelet
Namespace:         kube-system
Labels:            app.kubernetes.io/managed-by=prometheus-operator
                   app.kubernetes.io/name=kubelet
                   k8s-app=kubelet
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  RequireDualStack
IP Families:       IPv4,IPv6
IP:                None
IPs:               None
Port:              https-metrics  10250/TCP
TargetPort:        10250/TCP
Endpoints:         10.XXX.XX.XX:10250,10.XXX.XX.XX:10250,10.XXX.XX.XX:10250 + 5 more...
Port:              http-metrics  10255/TCP
TargetPort:        10255/TCP
Endpoints:         10.XXX.XX.XX:10255,10.XXX.XX.XX:10255,10.XXX.XX.XX:10255 + 5 more...
Port:              cadvisor  4194/TCP
TargetPort:        4194/TCP
Endpoints:         10.XXX.XX.XX:4194,10.XXX.XX.XX:4194,10.XXX.XX.XX:4194 + 5 more...
Session Affinity:  None
Events:            <none>

The ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: kps-kubelet
    app.kubernetes.io/instance: kps
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kps
    app.kubernetes.io/version: 44.3.0
    chart: kps-44.3.0
    heritage: Helm
    prometheus-system: "true"
    release: kps-system
  name: kps-system-kubelet
  namespace: kps-system
spec:
  attachMetadata:
    node: true
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    port: https-metrics
    relabelings:
    - action: replace
      sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    metricRelabelings:
    - action: drop
      regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
      sourceLabels:
      - __name__
    - action: drop
      regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
      sourceLabels:
      - __name__
    - action: drop
      regex: container_memory_(mapped_file|swap)
      sourceLabels:
      - __name__
    - action: drop
      regex: container_(file_descriptors|tasks_state|threads_max)
      sourceLabels:
      - __name__
    - action: drop
      regex: container_spec.*
      sourceLabels:
      - __name__
    - action: drop
      regex: .+;
      sourceLabels:
      - id
      - pod
    path: /metrics/cadvisor
    port: https-metrics
    relabelings:
    - action: replace
      sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    path: /metrics/probes
    port: https-metrics
    relabelings:
    - action: replace
      sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      app.kubernetes.io/name: kubelet
      k8s-app: kubelet

The resulting Configuration in prometheus:

- job_name: serviceMonitor/monitoring/kube-prometheus-stack-kubelet/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
...
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
    attach_metadata:
      node: true

Although the parameter is present in the Prometheus configuration, node metadata do not get attached.

What did you expect to see?

Node metadata attached to the scrape target.

What did you see instead? Under which circumstances?

No other solution found so far.

System information

Kube-Prometheus-Stack v44.3.0

Prometheus version

2.42.0

Prometheus configuration file

- job_name: serviceMonitor/monitoring/kube-prometheus-stack-kubelet/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
...
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
    attach_metadata:
      node: true

Alertmanager version

0.25.0

Alertmanager configuration file

not relevant

Logs

No logs related with the issue, not even in debug mode.

prometheus / prometheus

Node labels for endpoint role doesn't seem to work on services without any runtime in the cluster #11961