[receiver/prometheus] dns-sd-configs not adding instance name/job

[owenhaynes]

Component(s)

receiver/prometheus

What happened?

Description

When using dns-sd-configs to discover prometheus targets job and instance names are not populated causing the following error

2024-01-10T10:10:32.257Z        debug   scrape/scrape.go:1696   Unexpected error        {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "kafka", "target": "https://127.0.0.1:9273/metrics", "series": "kafka_consumer_group_offset{host=\"kafka\",name=\"foo\",partition=\"0\",topic=\"foo\"}", "error": "job or instance cannot be found from labels"}

I have tried using replace_labels to inject them, but the pipeline never gets that far before the error. But these should have been added at scrape time.

Steps to Reproduce

run otel-collector-contrib scrape dns-target

Expected Result

Being able to scrape using dns-sd-config

Actual Result

get metrics

Collector version

0.91.0

Environment information

Environment

OS: Ubuntu 22.04.2 LTS

OpenTelemetry Collector configuration

receivers:
  prometheus:
    config:
      scrape_configs:
      - job_name: 'kafka'
        scrape_timeout: 60s
        scrape_interval: 60s
        scheme: https
        metrics_path: "/metrics"
        honor_labels: false
        dns_sd_configs:
          - names:
              - kafaka
            type: A
            port: 9273
        tls_config:
          insecure_skip_verify: true

processors:
  batch:

extensions:
  health_check:
  pprof:
  zpages:

exporters:
  prometheus:
    endpoint: :9100
    add_metric_suffixes: false
    enable_open_metrics: false

service:
  extensions: [health_check, pprof, zpages]
  telemetry:
    logs:
      level: "debug"
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [prometheus]

Log output

2024-01-10T10:10:32.257Z        debug   scrape/scrape.go:1696   Unexpected error        {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "kafka", "target": "https://127.0.0.1:9273/metrics", "series": "kafka_consumer_group_offset{host=\"kafka\",name=\"foo\",partition=\"0\",topic=\"foo\"}", "error": "job or instance cannot be found from labels"}

Additional context

It would be useful for this error to be none debug as it actually a issue

[owenhaynes]

had rewrite rule to drop job & instance 🤕