Adding OpenTelemetry attributes to logs forwarded to output Loki

[kelko]

I have a FluentBit container receiving data from a Web API via name: opentelemetry input. Using name: stdout output I see, that the log is added to FluentBit in the structured form (including attributes/properties/...whatever-you-want-to-call-it...), not only the pure message.

e.g.:

[2] opentelemetry.0: [[1717162605.894517831, {"EnvName"=>"Development", "{OriginalFormat}"=>"Hosting environment: {EnvName}"}], {"message"=>"Hosting environment: Development"}]

Note the EnvName in the first [[. Now I want that output to Loki target. The fluent bit YAML I use:

pipeline:
    inputs:
        - name: opentelemetry
          listen: 0.0.0.0

    outputs:
        - name: loki
          match: '*'
          host: host.docker.internal
          port: 3100
          labels: job=my

Loki is a different container on same host. The connection itself does work, but Loki only receives the last part, the {"message"=>"Hosting environment: Development"} without the attributes. But I do know Loki is able to handle structured logs (e.g. formatted as JSON), I did it in a different project (without OTel in between).

Now the question is: How do I tell FluentBit to include the attributes it receives from OTel in the message it sends to Loki? Or can the Loki output be extended to also include those attributes, not only the log message?

I thought about using a "Content Modifier" processor between input & output. But if I understand the document correctly I can modify either body or attributes. But it didn't seem like I can take over key-value pairs from attributes into body.

(Bonus point if someone can tell me which tags I need to use in above configuration as match for the Loki output. Neither v1_logs, as mentioned in FluentBit docs, nor opentelemetry.proto.collector.log.v1.LogService_Export works. I had to use '*' to get anything matching. API sends OTel via gRPc)

[edsiper]

@kelko hi, thanks for raising this issue.

I think we will need to extend our Loki connector to auto-populate data that comes from OTLP. Would you please provide some input example of data and how that should be sent to Loki (in JSON) ?

[kelko]

@edsiper Sure. Simply said: Just copy all properties from attributes into body. But two concrete examples:

First is a default log message written by ASP.NET when starting up. It arrives at FluentBit like this (output from name: stdout output):

[2] opentelemetry.0: [[1717162605.894517831, {"EnvName"=>"Development"}], {"message"=>"Hosting environment: {EnvName}"}]

Expected log message in Loki:

{ 
  "message": "Hosting environment: {EnvName}",
  "EnvName": "Development"
}

---

Second an own structured log send, created inside a "traced" call, so it is automatically populated with some trace attributes as well.

Code in C# to create it:

[LoggerMessage(
        Message = "{Parameter} is not correctly validated before calling Calculator having value {Value}",
        Level = LogLevel.Critical)]
    internal static partial void LogInvalidParameter(
        this ILogger logger,
        string Parameter,
        double Value);

It arrives at FluentBit like this (output from name: stdout output):

[0] opentelemetry.0: [[1717920471.739041242, {"Parameter"=>"y", "Value"=>-nan, "SpanId"=>"faf50150bdd83352", "TraceId"=>"16d565057fbc799b5ea3d5a311efbef6", "ParentId"=>"fa9ebb7db97bc4cb", "ConnectionId"=>"0HN48BCLLFF4F", "RequestId"=>"0HN48BCLLFF4F:00000003", "RequestPath"=>"/trigger-log", "ActionId"=>"9e37d12b-d604-44e1-97c6-7148e74ee683", "ActionName"=>"OTel.API.Controllers.Foo ( (OTel.API)"}], {"message"=>"{Parameter} is not correctly validated before calling Calculator having value {Value}"}]

Expected log message in Loki:

{
  "message": "{Parameter} is not correctly validated before calling Calculator having value {Value}",
  "Parameter": "y",
  "Value": "NaN",
  "SpanId": "faf50150bdd83352", 
  "TraceId": "16d565057fbc799b5ea3d5a311efbef6",
  "ParentId": "fa9ebb7db97bc4cb",
  "ConnectionId": "0HN48BCLLFF4F",
  "RequestId":"0HN48BCLLFF4F:00000003",
  "RequestPath": "/trigger-log",
  "ActionId": "9e37d12b-d604-44e1-97c6-7148e74ee683", 
  "ActionName": "OTel.API.Controllers.Foo (OTel.API)"
}

[edsiper]

moving it to the next milestone

[r2d2k]

I had the same problem. Can`t use Fluent-bit metadata as labels for Loki.