When ingesting Prometheus metrics with the prometheusreceiver, all metrics are associated to the same Resource, which is simply the scraped target host (which happens to be localhost if you're using prometheusexecreceiver).
But the ingested Prometheus metrics may have a label that indicates that the metric relates to a different host (not the one where the Prometheus exporter is running), and this information cannot be leveraged and properly represented as a different Resource.
Without the proper Resource attachment, it is impossible for exporters to properly expose the Prometheus metrics (except the Prometheus exporters, which do not have such concept). Example: When exporting the metrics to Datadog (for example), only Resources (with a "host.name" label) are considered and displayed as Hosts.
Solution
This problem has already been solved with the groupbyattrs processor, which leverages the labels in a clever way to attach traces and logs to corresponding Resources. But this processor supports Traces and Logs signals only, unfortunately. Adding support for Metrics in the groupbyattrs processor would solve the problem very elegantly and would make Prometheus metrics ingestion much better translated into OTL metrics.
Alternatives
I've considered leveraging the metricstransform processor (which has a Group-By-Metric feature, but that doesn't allow using values coming from labels).
I've also considered the resourcedetection processor, but it doesn't process metrics and associates all metrics to the same Resource.
pmm-sumo
commented
2 years ago
Use Case
When ingesting Prometheus metrics with the prometheusreceiver, all metrics are associated to the same Resource, which is simply the scraped target host (which happens to be localhost if you're using prometheusexecreceiver).
But the ingested Prometheus metrics may have a label that indicates that the metric relates to a different host (not the one where the Prometheus exporter is running), and this information cannot be leveraged and properly represented as a different Resource.
Without the proper Resource attachment, it is impossible for exporters to properly expose the Prometheus metrics (except the Prometheus exporters, which do not have such concept). Example: When exporting the metrics to Datadog (for example), only Resources (with a "host.name" label) are considered and displayed as Hosts.
Solution
This problem has already been solved with the groupbyattrs processor, which leverages the labels in a clever way to attach traces and logs to corresponding Resources. But this processor supports Traces and Logs signals only, unfortunately. Adding support for Metrics in the groupbyattrs processor would solve the problem very elegantly and would make Prometheus metrics ingestion much better translated into OTL metrics.
Alternatives
I've considered leveraging the metricstransform processor (which has a Group-By-Metric feature, but that doesn't allow using values coming from labels).
I've also considered the resourcedetection processor, but it doesn't process metrics and associates all metrics to the same Resource.
More information
Example of Prometheus metrics being consumed:
...do not appear as 2 separate hosts in Datadog, as they are associated to
localhost:because Datadog simply creates Hosts based on OpenTelemetry Resources (with a proper
host.namelabel).