pod labels

[opvizordz]

Hi @mumoshu, great work - Thanks! Quick question - I couldn't manage to get the PodLabels working, when running the process exporter. I see the nodes, threads aso. but unfortunately, never the Pod names. Any hint or idea how to get that to work?

Thanks

[mumoshu]

@opvizordz Hey!

Oh, really?! I think I have never encountered that issue myself. Would you mind sharing your configuration, especially:

• how you've installed prometheus
• your prometheus configuration yaml(if possible
• whether your other metrics have pod name labels, or not

So that perhaps I can help investigate it

[opvizordz]

@mumoshu thanks for the lightning fast response. I've installed Prometheus using helm in a pretty default way. All metrics come in correct, just never with a pod name. Metric example - promtail runs in a pod:

namedprocess_namegroup_num_procs{app="prometheus-process-exporter",chart="prometheus-process-exporter-0.2.2",groupname="promtail",heritage="Tiller",instance="10.1.83.224:9100",job="kubernetes-service-endpoints",kubernetes_name="process-exporter-prometheus-process-exporter",kubernetes_namespace="default",kubernetes_node="juju-e1b2fa-2",release="process-exporter"}

another example that has the pod name, but that is node_exporter not process_exporter

kube_pod_container_resource_requests_cpu_cores{app="prometheus",chart="prometheus-8.9.0",component="kube-state-metrics",container="wordpress",heritage="Tiller",instance="10.1.83.164:8080",job="kubernetes-service-endpoints",kubernetes_name="eerie-snake-prometheus-kube-state-metrics",kubernetes_namespace="default",kubernetes_node="juju-e1b2fa-2",namespace="default",node="juju-e1b2fa-2",pod="my-wordpress-wordpress-6854b86fbc-7w9kc",release="eerie-snake"}

--

prometheus yaml:

apiVersion: v1
data:
  alerts: |
    {}
  prometheus.yml: |
    global:
      evaluation_interval: 1m
      scrape_interval: 1m
      scrape_timeout: 10s
    rule_files:
    - /etc/config/rules
    - /etc/config/alerts
    scrape_configs:
    - job_name: prometheus
      static_configs:
      - targets:
        - localhost:9090
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-apiservers
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: default;kubernetes;https
        source_labels:
        - __meta_kubernetes_namespace
        - __meta_kubernetes_service_name
        - __meta_kubernetes_endpoint_port_name
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes-cadvisor
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - job_name: kubernetes-service-endpoints
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: kubernetes_namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_service_name
        target_label: kubernetes_name
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: kubernetes_node
    - honor_labels: true
      job_name: prometheus-pushgateway
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      - action: keep
        regex: pushgateway
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_probe
    - job_name: kubernetes-services
      kubernetes_sd_configs:
      - role: service
      metrics_path: /probe
      params:
        module:
        - http_2xx
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_probe
      - source_labels:
        - __address__
        target_label: __param_target
      - replacement: blackbox
        target_label: __address__
      - source_labels:
        - __param_target
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels:
        - __meta_kubernetes_namespace
        target_label: kubernetes_namespace
      - source_labels:
        - __meta_kubernetes_service_name
        target_label: kubernetes_name
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: kubernetes_namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: kubernetes_pod_name

    alerting:
      alertmanagers:
      - kubernetes_sd_configs:
          - role: pod
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace]
          regex: default
          action: keep
        - source_labels: [__meta_kubernetes_pod_label_app]
          regex: prometheus
          action: keep
        - source_labels: [__meta_kubernetes_pod_label_component]
          regex: alertmanager
          action: keep
        - source_labels: [__meta_kubernetes_pod_container_port_number]
          regex:
          action: drop
  rules: |
    {}
kind: ConfigMap
metadata:
  creationTimestamp: "2019-03-30T09:46:25Z"
  labels:
    app: prometheus
    chart: prometheus-8.9.0
    component: server
    heritage: Tiller
    release: eerie-snake
  name: eerie-snake-prometheus-server
  namespace: default
  resourceVersion: "8622"
  selfLink: /api/v1/namespaces/default/configmaps/eerie-snake-prometheus-server
  uid: af54b514-52d0-11e9-b4fc-0050561f324c

[mumoshu]

@opvizordz Ah so it seems like we need to enhance process-exporter to include pod names as metric labels.

Perhaps it should be enhanced to work like this:

1. process-exporter collects all the processes running on the node
2. process-exporter collects all the linux pid namespaces the processes are in
3. process-expoter looks up containers and corresponding pods correspond to the linux pid namespaces
4. finally we have process-to-pod-name mapping that can be used for labelling...

[mumoshu]

Would you mind writing a feature request to process-exporter?

[opvizordz]

@mumoshu I thought the same, but wasn't entirely sure. Yes, I'll write a feature request. Thanks for your support!

[opvizordz]

done: https://github.com/ncabatoff/process-exporter/issues/90