Closed Daniel-Vaz closed 1 year ago
Related with https://github.com/prometheus-community/helm-charts/issues/2961
Using the prometheus-operator I'm trying to attach node metadata from a service that points to Kubelet.
The Kubelet Service:
# Manifest apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/managed-by: prometheus-operator app.kubernetes.io/name: kubelet k8s-app: kubelet name: kps-system-kubelet namespace: kube-system spec: clusterIP: None clusterIPs: - None internalTrafficPolicy: Cluster ipFamilies: - IPv4 - IPv6 ipFamilyPolicy: RequireDualStack ports: - name: https-metrics port: 10250 protocol: TCP targetPort: 10250 - name: http-metrics port: 10255 protocol: TCP targetPort: 10255 - name: cadvisor port: 4194 protocol: TCP targetPort: 4194 sessionAffinity: None type: ClusterIP # Describe Name: kps-system-kubelet Namespace: kube-system Labels: app.kubernetes.io/managed-by=prometheus-operator app.kubernetes.io/name=kubelet k8s-app=kubelet Annotations: <none> Selector: <none> Type: ClusterIP IP Family Policy: RequireDualStack IP Families: IPv4,IPv6 IP: None IPs: None Port: https-metrics 10250/TCP TargetPort: 10250/TCP Endpoints: 10.XXX.XX.XX:10250,10.XXX.XX.XX:10250,10.XXX.XX.XX:10250 + 5 more... Port: http-metrics 10255/TCP TargetPort: 10255/TCP Endpoints: 10.XXX.XX.XX:10255,10.XXX.XX.XX:10255,10.XXX.XX.XX:10255 + 5 more... Port: cadvisor 4194/TCP TargetPort: 4194/TCP Endpoints: 10.XXX.XX.XX:4194,10.XXX.XX.XX:4194,10.XXX.XX.XX:4194 + 5 more... Session Affinity: None Events: <none>
The ServiceMonitor:
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app: kps-kubelet app.kubernetes.io/instance: kps app.kubernetes.io/managed-by: Helm app.kubernetes.io/part-of: kps app.kubernetes.io/version: 44.3.0 chart: kps-44.3.0 heritage: Helm prometheus-system: "true" release: kps-system name: kps-system-kubelet namespace: kps-system spec: attachMetadata: node: true endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token metricRelabelings: - action: drop regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total) sourceLabels: - __name__ - action: drop regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total) sourceLabels: - __name__ - action: drop regex: container_memory_(mapped_file|swap) sourceLabels: - __name__ - action: drop regex: container_(file_descriptors|tasks_state|threads_max) sourceLabels: - __name__ - action: drop regex: container_spec.* sourceLabels: - __name__ - action: drop regex: .+; sourceLabels: - id - pod path: /metrics/cadvisor port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token path: /metrics/probes port: https-metrics relabelings: - action: replace sourceLabels: - __metrics_path__ targetLabel: metrics_path scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true jobLabel: k8s-app namespaceSelector: matchNames: - kube-system selector: matchLabels: app.kubernetes.io/name: kubelet k8s-app: kubelet
The resulting Configuration in prometheus:
- job_name: serviceMonitor/monitoring/kube-prometheus-stack-kubelet/0 honor_labels: true honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: https ... kubernetes_sd_configs: - role: endpoints kubeconfig_file: "" follow_redirects: true enable_http2: true namespaces: own_namespace: false names: - kube-system attach_metadata: node: true
Although the parameter is present in the Prometheus configuration, node metadata do not get attached.
Node metadata attached to the scrape target.
No other solution found so far.
Kube-Prometheus-Stack v44.3.0
2.42.0
0.25.0
not relevant
No logs related with the issue, not even in debug mode.
+
What did you do?
Related with https://github.com/prometheus-community/helm-charts/issues/2961
Using the prometheus-operator I'm trying to attach node metadata from a service that points to Kubelet.
The Kubelet Service:
The ServiceMonitor:
The resulting Configuration in prometheus:
Although the parameter is present in the Prometheus configuration, node metadata do not get attached.
What did you expect to see?
Node metadata attached to the scrape target.
What did you see instead? Under which circumstances?
No other solution found so far.
System information
Kube-Prometheus-Stack v44.3.0
Prometheus version
Prometheus configuration file
Alertmanager version
Alertmanager configuration file
Logs