prometheus.remote_write: external_labels does not override existing metrics

[532910]

What's wrong?

This:

pyroscope.write "default" {
    endpoint { url = "https://endpoint" }
    external_labels = { "instance" = constants.hostname }
}

replaces {"instance"="localhost:port"} with {"instance"="<hostname>"}

This:

prometheus.remote_write "default" {
    endpoint { url = "https://endpoint" }
    external_labels = { "instance" = constants.hostname }
}

does not.

Steps to reproduce

see above

System information

No response

Software version

grafana-agent-flow 0.39.2-1

Configuration

No response

Logs

No response

[tpaschalis]

Hey there, thanks for the report! 👋

The concept of external_labels in Prometheus is a global-level setting that is in use with many facets in the Prometheus ecosystem.

More specifically, when remote writing metrics, the behavior is indeed to not overwrite values that are already present https://github.com/prometheus/prometheus/blob/c6c8f63516741fd38ece5387254afee798cfc8d7/storage/remote/queue_manager.go#L974-L982

IMHO, this is for a good reason as it makes sure that external_labels cannot accidentally break the same pipeline run on two different Prometheus servers and we're following the same behavior on our Loki components as well.

So maybe here it's the Pyroscope implementation that should be changed. Any thoughts? cc @simonswine @cyriltovena

[532910]

1. behaviour must be the same (in prometheus and pyroscope)
2. it whould be nice to have a control to override external_labels

I have migrated from victoria metrics, and vmagent provides -remoteWrite.label which forces given labels.

[simonswine]

"Force overwriting" labels comes with the risk of accidental overlap of metrics. I think pyroscope needs to follow the other implementations here and it is a bug in pyroscope.write

[github-actions[bot]]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

[532910]

stale is not a reason to close issues

[rfratto]

Hi there :wave:

On April 9, 2024, Grafana Labs announced Grafana Alloy, the spirital successor to Grafana Agent and the final form of Grafana Agent flow mode. As a result, Grafana Agent has been deprecated and will only be receiving bug and security fixes until its end-of-life around November 1, 2025.

To make things easier for maintainers, we're in the process of migrating all issues tagged variant/flow to the Grafana Alloy repository to have a single home for tracking issues. This issue is likely something we'll want to address in both Grafana Alloy and Grafana Agent, so just because it's being moved doesn't mean we won't address the issue in Grafana Agent :)

[steveteuber]

This configuration works for me without any problems (Grafana Alloy v1.2.0):

prometheus.exporter.self "alloy" {}

prometheus.scrape "alloy" {
  targets    = prometheus.exporter.self.alloy.targets
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "..."

    write_relabel_config {
      source_labels = ["job"]
      target_label  = "job"
      replacement   = "my-prefix/my-job-name"
    }
  }
}