K8s Discovery - Non pyroscope annotations

[syepes]

I have been trying to use the pull method without using the predefined pyroscope.io/ annotations and simply using the OOTB ones and it does not seam to works.

Is there some sort of hard coded think that only allow us to use the pyroscope.io/ annotations?

https://github.com/pyroscope-io/pyroscope/blob/main/examples/golang-pull/kubernetes/values.yaml#L14-L20

pyroscopeConfigs:
  analytics-opt-out: true
  log-level: debug
  storage-path: "/var/lib/pyroscope"
  scrape-configs:
    - job-name: "kubernetes-pods"
      enabled-profiles: [cpu, mem, goroutines, mutex, block]
      kubernetes-sd-configs:
        - role: pod
      relabel_configs:
        - action: keep
          regex: .*true.*
          source_labels:
            - __meta_kubernetes_pod_label_mon_scrape
            - __meta_kubernetes_pod_annotation_mon_scrape
        - action: keep
          regex: .*http.*|health
          source_labels:
            - __meta_kubernetes_pod_container_port_name
        - action: replace
          source-labels:
            - __meta_kubernetes_pod_container_name
          target-label: __name__

Logs

me="2022-10-03T12:40:04.017785"  level=error msg="creating target" file=" scrape/scrape.go:237"  component=scrape-manager error="instance 3 in group  pod/vm/vmstorage-0: no app name"

[kolesnikovae]

Hello @syepes. Thank you for reporting the issue!

It looks like a bug. The error states that __name__ label is an empty string:

        - action: replace
          source-labels:
            - __meta_kubernetes_pod_container_name
          target-label: __name__ 

I'll see what can be wrong. May I ask you to temporary set some value to this label and write __meta_kubernetes_pod_container_name to another one, e.g:

        - source-labels: [__meta_kubernetes_pod_name]
          action: replace
          target-label: kubernetes_pod_name

Then, kubernetes_pod_name label should appear in the pyroscope UI, or not, if it is empty, indeed.

Also, just in case: could you post the versions of pyroscope and k8s please?

---

Is there some sort of hard coded think that only allow us to use the pyroscope.io/ annotations?

No, there is no such thing: pyroscope.io/ prefix is just for relabeling

[syepes]

Thanks for looking at it.

I am using the version 0.29.0 and k8s v1.20.14. Even if I use the kubernetes_pod_name it still does not scrape the profile as there is a check (see below) that does not allow if no app name is defined.

https://github.com/pyroscope-io/pyroscope/blob/67227bfdeab12cda7920255cd9413c0aa3c8db0a/pkg/scrape/target.go#L268-L270 (No app name check)

https://github.com/pyroscope-io/pyroscope/blob/67227bfdeab12cda7920255cd9413c0aa3c8db0a/pkg/scrape/model/labels.go#L93 (model.AppNameLabel <- name)

[kolesnikovae]

Right, at least __name__ and __address__ labels must be present.

Even if I use the kubernetes_pod_name it still does not scrape the profile as there is a check (see below) that does not allow if no app name is defined.

It's interesting, __meta_kubernetes_pod_name shouldn't be empty. I'm curious if pyroscope reported the same error: I just noticed that __address__ label is missing as well:

        - source-labels:
            [__address__]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:6060 # Default pprof port
          target-label: __address__

[syepes]

Just tried the below config that contains both __address__ and __name__ and I am still getting the no app name errors

      relabel_configs:
        - action: keep
          regex: .*true.*
          source_labels:
            - __meta_kubernetes_pod_label_mon_scrape
            - __meta_kubernetes_pod_annotation_mon_scrape
        - action: replace
          regex: (.+);(\d+)
          replacement: $1:$2
          source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_pod_annotation_mon_port]
          separator: ;
          target_label: __address__
        - action: replace
          source-labels:
            - __meta_kubernetes_pod_container_name
          target-label: __name__
        - action: replace
          regex: (.+)
          source_labels:
            - __meta_kubernetes_pod_name
          target_label: instance
        - action: replace
          regex: (.+)
          source_labels:
            - __meta_kubernetes_pod_node_name
          target_label: node
        - action: replace
          regex: (.+)
          source_labels:
            - __meta_kubernetes_pod_host_ip
          target_label: node_ip
        - action: replace
          regex: (.+)
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace

[kolesnikovae]

Thank you @syepes for your time! I've prepared an example working configuration. There are couple of considerations:

• using pod name as application name might be not what you want: names are generated every time you deploy a new version (except StatefulSets). This will create a new item in the applications dropdown menu in UI for every pod version.
• pyroscope uses kebab-style yaml in contrast to camel_case in prometheus.

We recommend using an explicit application name in the configuration. The example below re-uses existing app.kubernetes.io/name pod label.

Scrape configuration:

  scrape-configs:
    - job-name: 'kubernetes-pods'
      enabled-profiles: [cpu, mem]
      kubernetes-sd-configs:
        - role: pod
      relabel-configs:
        # Filter-out only pods eligiable for scraping.
        - source-labels:
            [__meta_kubernetes_pod_annotation_mon_scrape]
          action: keep
          regex: true
        # Figure out the target address.
        - source-labels:
            [__address__, __meta_kubernetes_pod_annotation_mon_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target-label: __address__
        # Take the app name from 'app.kubernetes.io/name' pod label.
        - source-labels: [ __meta_kubernetes_pod_label_app_kubernetes_io_name ]
          action: replace
          target-label: __name__

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hotrod-golang
spec:
  selector:
    matchLabels:
      app: hotrod-golang
  template:
    metadata:
      labels:
        app: hotrod-golang
        app.kubernetes.io/name: hotrod-golang
      annotations:
        mon_scrape: 'true'
        mon_port: '6060'
    spec: {} # omitted

[syepes]

Thanks for the help, this is now working with the last version v0.30.0 and config fixes