Closed syepes closed 1 year ago
Hello @syepes. Thank you for reporting the issue!
It looks like a bug. The error states that __name__
label is an empty string:
- action: replace
source-labels:
- __meta_kubernetes_pod_container_name
target-label: __name__
I'll see what can be wrong. May I ask you to temporary set some value to this label and write __meta_kubernetes_pod_container_name
to another one, e.g:
- source-labels: [__meta_kubernetes_pod_name]
action: replace
target-label: kubernetes_pod_name
Then, kubernetes_pod_name
label should appear in the pyroscope UI, or not, if it is empty, indeed.
Also, just in case: could you post the versions of pyroscope and k8s please?
Is there some sort of hard coded think that only allow us to use the pyroscope.io/ annotations?
No, there is no such thing: pyroscope.io/
prefix is just for relabeling
Thanks for looking at it.
I am using the version 0.29.0 and k8s v1.20.14.
Even if I use the kubernetes_pod_name
it still does not scrape the profile as there is a check (see below) that does not allow if no app name is defined.
https://github.com/pyroscope-io/pyroscope/blob/67227bfdeab12cda7920255cd9413c0aa3c8db0a/pkg/scrape/target.go#L268-L270 (No app name check)
https://github.com/pyroscope-io/pyroscope/blob/67227bfdeab12cda7920255cd9413c0aa3c8db0a/pkg/scrape/model/labels.go#L93 (model.AppNameLabel <- name)
Right, at least __name__
and __address__
labels must be present.
Even if I use the
kubernetes_pod_name
it still does not scrape the profile as there is a check (see below) that does not allow if no app name is defined.
It's interesting, __meta_kubernetes_pod_name
shouldn't be empty. I'm curious if pyroscope reported the same error: I just noticed that __address__
label is missing as well:
- source-labels:
[__address__]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:6060 # Default pprof port
target-label: __address__
Just tried the below config that contains both __address__
and __name__
and I am still getting the no app name
errors
relabel_configs:
- action: keep
regex: .*true.*
source_labels:
- __meta_kubernetes_pod_label_mon_scrape
- __meta_kubernetes_pod_annotation_mon_scrape
- action: replace
regex: (.+);(\d+)
replacement: $1:$2
source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_pod_annotation_mon_port]
separator: ;
target_label: __address__
- action: replace
source-labels:
- __meta_kubernetes_pod_container_name
target-label: __name__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_name
target_label: instance
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_node_name
target_label: node
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_host_ip
target_label: node_ip
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
Thank you @syepes for your time! I've prepared an example working configuration. There are couple of considerations:
kebab-style
yaml in contrast to camel_case
in prometheus.We recommend using an explicit application name in the configuration. The example below re-uses existing app.kubernetes.io/name
pod label.
Scrape configuration:
scrape-configs:
- job-name: 'kubernetes-pods'
enabled-profiles: [cpu, mem]
kubernetes-sd-configs:
- role: pod
relabel-configs:
# Filter-out only pods eligiable for scraping.
- source-labels:
[__meta_kubernetes_pod_annotation_mon_scrape]
action: keep
regex: true
# Figure out the target address.
- source-labels:
[__address__, __meta_kubernetes_pod_annotation_mon_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target-label: __address__
# Take the app name from 'app.kubernetes.io/name' pod label.
- source-labels: [ __meta_kubernetes_pod_label_app_kubernetes_io_name ]
action: replace
target-label: __name__
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hotrod-golang
spec:
selector:
matchLabels:
app: hotrod-golang
template:
metadata:
labels:
app: hotrod-golang
app.kubernetes.io/name: hotrod-golang
annotations:
mon_scrape: 'true'
mon_port: '6060'
spec: {} # omitted
Thanks for the help, this is now working with the last version v0.30.0 and config fixes
I have been trying to use the
pull
method without using the predefinedpyroscope.io/
annotations and simply using the OOTB ones and it does not seam to works.Is there some sort of hard coded think that only allow us to use the
pyroscope.io/
annotations?https://github.com/pyroscope-io/pyroscope/blob/main/examples/golang-pull/kubernetes/values.yaml#L14-L20
Logs