loki.process option to flatten structured_metadata for otel attributes

Request

in loki.process, structured_metadata should optionally flatten a nested value, using the keys of the nested value as metadata keys.

For example:

An inbound message structured like this:

{
  "body": "log message",
  "attributes": {
      "foo": "bar",
      "abc": "xyz"
  }
}

Using configuration like this:

loki.process "logs_service" {
        stage.json {
                expressions = {"body" = "body", "attributes" = "attributes"}
        }

        stage.structured_metadata {
                values = {
                        attributes = "attributes",
                }
                flatten = true
        }

        stage.output {
                source = "body"
        }

Should produce output like this:

{ "line": "log message", "fields": { "foo": "bar", "abc": "xyz"}}

Alternatively, if there was a different value to set the loki.format hint to that would use the body as the log payload and set other specified keys as metadata, that would work as well.

Use case

I am using the otelcol.receiver to forward logs to Loki using otelcol.exporter.loki. I am using the loki.resource.labels hint to set certain resource attributes as Loki labels. I would like to set resource and log attributes as Loki log metadata.

Out of the box, with the default json setting of json the otelcol.exporter.loki will pass the Otel Log as something like the following, which is not really well suited for my purposes:

{
  "body": "log body",
  "attributes: {
        "logline_attribute": "a"
  },
   "resources": {
         "resource_attribute_a": "val"
  }
}

I can set the otelcol.exporter.loki hint to raw, which will send the body as the log payload, which is closer to what I want, but I lose the resource and log attributes in the process.

What is the closest to idea is the following:

loki.process "logs_service" {
        stage.json {
                expressions = {"body" = "body", "attributes" = "attributes", "resources" = "resources", "instrumentation_scope" = "i
nstrumentation_scope"}
        }

        stage.structured_metadata {
                values = {
                        attributes = "attributes",
                        resources  = "resources",
                }
        }

        stage.output {
                source = "body"
        }

This sets the log and resource attributes as values to metadata values, which is better than nothing, but I'm not aware of a way in LogQL to parse metadata fields as JSON.

grafana / alloy

loki.process option to flatten structured_metadata for otel attributes #1173

Request

Use case