grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.48k stars 221 forks source link

Prometheus relabel within prometheus.scrape component for params in Flow mode #376

Open Whyeasy opened 1 year ago

Whyeasy commented 1 year ago

Request

Within the agent in static mode (or Prometheus) it is possible to access provided params inside of the relabel config like this:

scrape_configs:
  - job_name: blackbox
    metrics_path: /probe
    scrape_interval: 60s
       params:
         module: [<module_name>]
         target:
           [<URL>]
       relabel_configs:
         - source_labels: [__param_module]
           target_label: module

With Flow and the components that come with it, it is currently not possible to access these __param_ values. They are added within a prometheus.scrape component. Which discards any __ labels when it forwards to a remote write endpoint.

This is the current configuration used to work around this issue:

// Discover the blackbox exporter pods
discovery.kubernetes "k8s_pods_blackbox"{
  role = "pod"

  namespaces {
    names = [
        "grafana-agent",
    ]
  }
}

discovery.relabel "blackbox_exporter" {
  targets = discovery.kubernetes.k8s_pods_blackbox.targets

  rule {
    source_labels = [
      "__meta_kubernetes_pod_label_app_kubernetes_io_name",
    ]
    action = "keep"
    regex = "prometheus-blackbox-exporter"
  }

  rule {
    source_labels = [
      "__address__",
      "__meta_kubernetes_service_annotation_prometheus_io_port",
    ]
    regex = "(.+):(?:d+);(d+)"
    replacement = "${1}:${2}"
    target_label = "__address__"
  }

  rule {
    source_labels = [
      "__meta_kubernetes_pod_name",
    ]
    target_label = "pod"
  }

  rule {
    replacement = "blackbox_exporter"
    target_label = "app"
  }
}

discovery.relabel "blackbox_exporter_<module>" {
  targets = discovery.relabel.blackbox_exporter.output

  rule {
    source_labels = [
      "instance",
    ]
    target_label = "instance"
    replacement = "<URL>"
  }

  rule {
    replacement = "<module_name>"
    target_label = "module"
  }
}

prometheus.scrape "blackbox_<module>" {
  targets    = discovery.relabel.blackbox_exporter_<module>.output
  forward_to = [prometheus.remote_write.prom.receiver]
  scrape_interval = "60s"
  metrics_path = "/probe"
  params = {
    "module" = ["<module_name>"],
    "target" = ["<URL>"],
  }
}

Use case

The workaround is to create a separate discovery.relabel component for each different module/target. Would be great to have a way to have access to the __param_module label inside or after the prometheus.scrape component. This will simplify the config.

github-actions[bot] commented 1 year ago

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

ghobadimhd commented 3 months ago

Hello. I have a same problem here. Is there updates on this?