search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.92k stars 2.28k forks source link

[exporter/prometheus] shouldn't `resource_to_telemetry_conversion.enabled` be `true` by default? #35286

Open mathnogueira opened 5 days ago

mathnogueira commented 5 days ago

Component(s)

exporter/prometheus, exporter/prometheusremotewrite

Describe the issue you're reporting

I was working on a local stack for retrieving metrics from Kubernetes and also receive metrics from our applications and publish them to our local Grafana. Everything was working fine but all metrics were strange because they had no labels (i.e. no attributes were sent to grafana).

After 4 days debugging the connections between our k8s scrapper collectors, our main collector, and our prometheus server, I discovered I was missing the resource_to_telemetry_conversion.enabled: true option in my prometheus exporter config.

27839 was useful to discover I was missing that configuration. It's not the first time I had this problem, last year I had the same issue with the prometheusremovewrite exporter and I had forgotten about it. I was wondering why this flag is set as false as its default value. Couldn't this be set as true as this is crucial to have metrics working properly with prometheus?

github-actions[bot] commented 5 days ago

Pinging code owners:

dashpole commented 4 days ago

This is something we are considering, but have chosen not to do right now because resource can contain a huge number of attributes, which are often very long (e.g. process.command_args).

Instead, we provide a metric named target_info, which can be joined with your metrics to achieve similar behavior.

cc @gouthamve, since this is related to recent proposals around entities.

ArthurSens commented 3 days ago

@mathnogueira, please take look at Prometheus' guide about how to use target_info to add resource attributes to your queries: https://prometheus.io/docs/guides/opentelemetry/#including-resource-attributes-at-query-time

There is also effort on OTel's side to improve compatibility with Prometheus. This OTEP is a good step forward, but it will take a while for both communities to come up with implementations

mathnogueira commented 3 days ago

@ArthurSens thanks for the reference. This will definitely help!