Feat: Add page detailing possibilities and best practices for communication with Prometheus

[ywwg]

What needs to be changed? Describe the update that is required.

There are currently 4 ways Otel can speak with Prometheus (and I'm not 100% sure I have the terminology right for all of these -- the final list will be correct):

• Otel Collector prometheusexporter for scraping by Prometheus
• Otel Collector prometheusremotewriteexporter for direct push to Prometheus
• Otel native export to Prometheus Otel endpoint
• Prometheus export to Otel
• Possibly collector receivers as well?

The plethora of options is confusing, and it would be useful to have a document detailing all of the possibilities and their pros and cons, as well as a clear recommendation for the "best" way to get data from Otel to Prometheus

What is the name + path of the page that needs changed? The relative path and page title where you found a problem.

I am not sure of the best place to put this doc -- in the Collector section? Open to suggestions

[tiffany76]

Thanks for the suggestion, @ywwg!

@open-telemetry/prometheus-interoperability and @open-telemetry/collector-approvers, PTAL!

Note to the @open-telemetry/docs-approvers: I tried searching the website for prometheus to see if we have anything close to this in the docs. The search result page came back blank.

[mx-psi]

Makes sense to me to add this. I think this should not be in the Collector section though, I feel like newbies don't necessarily need to know that the Collector is the recommended way of doing this.

[svrnm]

As of today we do not have a section to capture that, if we move forward with rolling out a "Learn"/"Tutorial" section based on the getting started proposal, that would be the perfect place for it, but for now I would say "wherever it fits, let's have it"

[dashpole]

I'm very supportive. We discussed this at the Prometheus interoperability sig on 11/20/24, and agreed that this would be very helpful.

[jpkrohling]

cc @ArthurSens

[ArthurSens]

If we count all possibilities, you could mention the usage of Prometheus SDK + Collector's Prometheus Receiver (it scrapes Prometheus endpoints) + Any of the three exporter options. You could also switch Prometheus SDK with OTel SDK and multiple all options again 😬. I'm not sure if we want to mention every possibility though.

[svrnm]

Let's start small and grow from there :-)

[ArthurSens]

I am dumping some info from previous experiences:

If you are only doing Prometheus metrics, you don't need OTLP at all. Use Prometheus SDK and Prometheus Server/Agent for scraping. Prometheus is optimized for this scenario.

OTLP is the way forward for SDKs if you need multiple signals or are doing metrics beyond Prometheus. Then, the doubt is whether you need a collector or not. You'll need a collector if:

• If you need data preprocessing, like removing/updating resource attributes.
• If your app uses Deltas, you'll need the deltatocumulative processor (at least until Delta support lands in Prometheus).
• If you want to limit data points per minute, you need the interval processor.
• If you are sensitive to network payload size, use a collector to transform OTLP into PRW and use PRW exporter.
• If you need any kind of load distribution, use a load balancer processor with a multi-collector architecture.

If you don't need any of those, send OTLP straight to Prometheus' OTLP endpoint :)

Maybe I missed something; I didn't spend too much time thinking here 😬