[exporter/loki] Auto convert OpenTelemetry ResourceAttributes to log labels

[fkalinowski]

Component(s)

exporter/loki

Is your feature request related to a problem? Please describe.

The Resource Attributes are very interesting to use as labels in the log events.

With the current implementation, I need to declare an explicit mapping in my collector's configuration for each Resource Attribute. This approach requires to reconfigure my opentelemetry-collector each time an application introduces a new Resource Attribute (available in SemConv or not).

Moreover due to https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/14113 I need to declare each ResourceAttribute twice - which can be tedious.

Describe the solution you'd like

Since Loki's backend is largely inspired by the Prometheus backend (Loki uses Mimir implementation of Prometheus compliant backend), that seems natural to apply the same constraint and have the same feature sets than exporter/prometheusremotewriteexporter.

That would be great if the feature resource_to_telemetry_conversion could be ported into the lokiexporter.

Describe alternatives you've considered

No response

Additional context

@gouthamve https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/14113#issuecomment-1456263808

[github-actions[bot]]

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @kovrus @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

cc @mar4uk

[mar4uk]

I think we can go for it. Adding resource_to_telemetry_conversion will also solve another issue https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/2290

So basically, if users trust their downstream configurations and keep the number of resource attributes minimal, we can allow users to promote that attributes to labels automatically. The fact that we have resource_to_telemetry_conversion for exporter/prometheusremotewriteexporter makes adding the same option to loki exporter nice to have.

We can add a warning to the description of this option in README: use with cautions since a high number of resource attributes can lead to high cardinality that will affect exploring such logs performance in Loki

@kovrus, @gouthamve

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @kovrus @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mar4uk]

let's keep this issue for a little bit longer

[kago-dk]

Were you still planning to implement resource_to_telemetry_conversion in the Loki Exporter?

[mar4uk]

Yes, it would be good to have

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mar4uk]

still relevant

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[sathieu]

This issue is still relevant.

But another road will be possible in the next loki version: https://grafana.com/docs/loki/next/send-data/otel/

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

@sandeepsukhani, do we have a logic like that already on the Loki side?

[sandeepsukhani]

@sandeepsukhani, do we have a logic like that already on the Loki side?

Hey @jpkrohling, we have this logic documented. The code for converting otlp format to loki format can be found here.

[jpkrohling]

@mar4uk, do you think the logic can be reused?

[mar4uk]

I think so. The logic could be like this: if resource_to_telemetry_conversion is set to true then we convert the same list of resource attributes as the native Loki OTLP endpoint does https://grafana.com/docs/loki/next/send-data/otel/#format-considerations to labels:

• cloud.availability_zone
• cloud.region
• container.name
• deployment.environment
• k8s.cluster.name
• k8s.container.name
• k8s.cronjob.name
• k8s.daemonset.name
• k8s.deployment.name
• k8s.job.name
• k8s.namespace.name
• k8s.pod.name
• k8s.replicaset.name
• k8s.statefulset.name
• service.instance.id
• service.name
• service.namespace

All the rest resource attributes will be sent as structured metadata. Does it sound good?

[jpkrohling]

Yes, sounds good to me!

[mhulscher]

I found this issue because I am also looking to store attributes as structured metadata. After the seeing the (static) list of converted attributes I was a little skeptical of the proposed solution. However, the docs clearly mention that it will be possible to extend or even replace this list. That would be amazing!

Edit: for example, host.name is missing from the list. Which would be a useful label to have if you are collecting telemetry from a VM!

[mk-raven]

Also interesting about this issue. Any updates?

[crutonjohn]

this is a feature i'd like as well. been spending hours and hours trying to get otelcol to send "native" loki labels. a plug-and-play option would be awesome.

[jpkrohling]

@mhulscher , @mk-raven , @crutonjohn , given that Loki 3.0 supports OTLP natively, I'm interested in knowing whether you still see the need for the Loki exporter.

[mk-raven]

@jpkrohling hello! We needed this exporter because some service, use loki api for store telemetry data in Clickhouse DB. There are products like qryn, oteldb and other could integrate loki api for natively support logs querying in Grafana UI. Without manage custom plugins. Its great feature for many companies and teams.

[jpkrohling]

qryn, oteldb

A quick look into both makes me believe they support OTLP natively as well.

[mk-raven]

Its true, but Grafana UI doesn't support query in otlp format. Its has Clickhouse plugin, but for many people its hard to using. Better use natively LogQL, PromQL and TraceQL

[jpkrohling]

Better use natively LogQL, PromQL and TraceQL

Ingesting data in OTLP doesn't mean you have to query data in OTLP (even if there was one query language for OTLP). You should still be able to send data in via OTLP and query them via LogQL, as we do with Grafana Cloud :-)

[mk-raven]

But from otlp logs there no query filters in Grafana, if You use LogQL and Loki datasource.

[jpkrohling]

It's certainly possible to use LogQL to query a Loki data source containing logs that were ingested via OTLP.