GMP: Support for Prometheus Remote Write

[edwardcrupi]

Hi Folks,

My team is considering moving over to prometheus-engine and is currently using prometheus-operator as that allows us to use the remote-write-receiver flag. Can anyone provide a view as to when a feature like the remote-web-receive flag used by vanilla prometheus might be exposed by prometheus-engine?

Kind regards, Edward

[bwplotka]

Hi!

Thanks for reaching out on this and trying our work! 💪🏽

Enabling remote write receiving on our drop-in Prometheus binary is an interesting idea. Could you share more information about your use case? Why would it be useful?

Generally, the remote-write-receiver functionality in vanilla Prometheus is still experimental and, as a Prometheus maintainer, I can tell you that it was not designed for high volume data, nor for "proxying" from one remote write to another remote endpoint. Thus it hasn't been used very often in our experience.

Our drop-in Prometheus binary supports this flag but is not tested and probably should be disabled. For example, while it would ingest data, there might be issues with exporting such metrics further into GMP. Ofc, we could consider improving this long term if there is a solid use case. Furthermore, our operator (part of prometheus-engine) does not support adding such a flag on a fully managed setup on GKE. Outside, in theory, you could customize it.

Side questions, if you don't mind:

1. By moving over to prometheus-engine, do you mean using managed collection (using prometheus-engine's operator on GKE or outside) or self-deployed collection (only using our Prometheus drop-in binary which integrates with GMP offering)?
2. What motivates you to use prometheus-engine? Is it integration with GMP or something else? (Just getting feedback. 🤗)

[parkedwards]

im not the OP, but i can share a use case. basically, if you're using prometheus (and GMP to house the collections and handle collecting/rule evaluations), you probably like the idea of aggregating multiple data sources in GMP

Grafana's tracing product, Tempo, has functionality to report metrics from the ingested distributed traces. however, they only support pushing metrics via remote-write, instead of scraping

another of Grafana's product, Loki, which is an alternative to an ELK stack for logging, allows you to generate metrics from log statistics. however, these metrics are also only writeable through push, which requires a remote write endpoint

[pintohutch]

Thanks for the use cases @parkedwards. I think the ultimate fix is for the GMP backend to support PRW.

I don't think it makes a lot of sense to use this feature with managed collection as there, Prometheus is a node agent and metric data is sharded across the agents and only temporarily available locally before it's written to Google Cloud Monitoring (GCM).

You could use the standalone binary @bwplotka mentioned, but remote write strips TYPE information, which would cause ingestion issues to GCM.

[pintohutch]

We can leave this issue open to track GMP support for PRW generally.

[shinebayar-g]

Another use case. k6 only supports remote write endpoint. https://k6.io/docs/results-output/real-time/prometheus-remote-write/

[pravinbhat]

AstraDB (used by many GCP customers) also only supports destinations with remote-write-receiver

[nategoethel]

Our use case would be to export metrics to Coralogix.

[lyanco]

FWIW Managed Collection can be configured to use remote write to tee data to a third-party destination: https://github.com/GoogleCloudPlatform/prometheus-engine/blob/v0.12.0/doc/api.md#exportspec

In terms of GMP being configured to receive remote write, we've worked with the community to finalize the spec for Remote Write v2, which solves all the compatibility issues that exist today with remote write. Expect the receiver and client to both be ready in 2025.

[nategoethel]

FWIW Managed Collection can be configured to use remote write to tee data to a third-party destination: https://github.com/GoogleCloudPlatform/prometheus-engine/blob/v0.12.0/doc/api.md#exportspec

Great! Thanks for the link. I'll look into this. We don't need to support remote write to GMP, just from GMP to Coralogix.