Closed paebersold-tyro closed 1 day ago
Pinging code owners:
receiver/prometheus: @Aneurysm9 @dashpole
See Adding Labels via Comments if you do not have permissions to add labels yourself.
Can you set the log level of the collector to debug to see the detailed error message for why the scrape failed?
I think it should be:
service:
logs:
level: DEBUG
Hello, debug log output (seems the empty instance label may be the issue as suspected)
2024-06-27T01:40:49.045Z debug scrape/scrape.go:1650 Unexpected error {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "test-pushgateway", "target": "http://10.18.67.95:9091/metrics", "series": "app_platform_attestation{feature=\"coredns\",instance=\"\",job=\"cluster\",team=\"bob\",test=\"TestCoreDNSNameResolution\"}", "error": "job or instance cannot be found from labels"}
2024-06-27T01:40:49.045Z debug scrape/scrape.go:1346 Append failed {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "test-pushgateway", "target": "http://10.18.67.95:9091/metrics", "error": "job or instance cannot be found from labels"}
2024-06-27T01:40:49.045Z warn internal/transaction.go:125 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1719452449041, "target_labels": "{__name__=\"up\", instance=\"10.18.67.95:9091\", job=\"test-pushgateway\"}"}
This should've been fixed by https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/33565. Can you try upgrading to v0.103.0?
Thank you for that, 0.103.0 fixed the issue.
Component(s)
receiver/prometheus
What happened?
Description
Scraping a Prometheus pushgateway with
honor_labels: true
results in a scrape endpoint failure. Suspect this is due to the scrape metrics having both instance and jobs labels (from https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/15239) but would like clarification that this is the problem. Also is there any work around (other than settinghonor_labels: false
). Attempted doing a label drop with metric_relabel_config but that did not work.Steps to Reproduce
Prometheus receiver config
Expected Result
Endpoint is scraped, job and instances labels from the pushgateway are used.
Actual Result
Endpoint scrape failure (see log message below)
Collector version
0.102.0
Environment information
Environment
OS: Kubernetes 1.29
OpenTelemetry Collector configuration
Log output
Additional context
sample of metrics that are returned from the pushgateway