grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.47k stars 221 forks source link

Loki OTLP ingestion doesn't properly set labels with otelcol.processor.attributes #2064

Closed lieberlois closed 2 weeks ago

lieberlois commented 2 weeks ago

What's wrong?

Hey everybody,

I'm trying to specify labels when ingesting logs into Loki via the otelcol.exporter.otlphttp exporter component. I want to "relabel" the service.name resource attribute to an attribute app and then add it to the loki.resource.labels or loki.attribute.labels (not sure which one is right here, i tried both). In the steps to reproduce you can see the code. For some reason, the label is not added. This is the output in the livedebugging feature (actual name in Str(...) omitted in the screenshot):

Image

I expected this to work since many other people (and also the docs) do it this way. It's also pretty common in the upstream OTEL Collector, as can be seen here.

I got it working by using another component like this first, but I dont think both should be necessary:

    otelcol.processor.transform "default" {
      log_statements {
        context = "resource"
        statements = [
          "set(attributes[\"app\"], attributes[\"service.name\"])",
        ]
      }

      output {
        logs   = [otelcol.processor.attributes.default.input]
      }
    }

Steps to reproduce

Add this otelcol component configuration to your alloy config.

    otelcol.processor.attributes "default" {
      // Extract app name from service.name attribute
      action {
        key = "app"
        from_attribute = "service.name"
        action = "upsert"
      }

      // Generate loki labels from OpenTelemetry attributes
      action {
        key = "loki.attribute.labels"
        action = "insert"
        value = "service.name,app"  // comma separated list
      }

      output {
        logs = [otelcol.processor.batch.default.input]
      }
  }

System information

No response

Software version

Grafana Alloy Helm Chart v0.9.1

Configuration

Logs

lieberlois commented 2 weeks ago

Issue on my side. The docs dont clearly state that you have to disable the defaults for OLTP in the loki config itself.