How to remove duplicate & non relevant attributes from the Otlp Log record before it is exported to OTLP log collector destination

[madhub]

Question

How to remove duplicate & non relevant attributes from the Otlp Log record before it is exported to OTLP log collector destination

1. Duplicate attributes : Open telemetry Log data model contains LogRecord.TraceId & LogRecord.SpainId attributes but same attributes also added LogRecord.attributes collection.
2. non relevant attributes : LogRecord.attributes collection contains ActionId, ActionName, ConnectionId,

I tried adding my own LogProcessorusing an derived implementation of BaseProcessor<LogRecord>. it does not work because LogRecord.attributes does not have these values when Otel library calls my implementation

public class DemoLogProcessor : BaseProcessor<LogRecord>{
   public override void OnEnd(LogRecord data) {
      // add logic to remove ActionId, ActionName, ConnectionId , TraceId and SpainId  from 
     // LogRecord.Attributes  Collection by creating new `IReadOnlyList<KeyValuePair<string, object?>` out of `LogRecord.Attributes`
   }   
}

Sample json log for reference

{
    "resourceLogs": [
        {
            "resource": {
                "attributes": [
                    {
                        "key": "service.name",
                        "value": {
                            "stringValue": "OtelLokiLogTest"
                        }
                    },
                    {
                        "key": "service.instance.id",
                        "value": {
                            "stringValue": "7cf6b018-0330-41f6-9846-4d5d69a6571e"
                        }
                    },
                    {
                        "key": "telemetry.sdk.name",
                        "value": {
                            "stringValue": "opentelemetry"
                        }
                    },
                    {
                        "key": "telemetry.sdk.language",
                        "value": {
                            "stringValue": "dotnet"
                        }
                    },
                    {
                        "key": "telemetry.sdk.version",
                        "value": {
                            "stringValue": "1.6.0"
                        }
                    }
                ]
            },
            "scopeLogs": [
                {
                    "logRecords": [
                        {
                            "timeUnixNano": "1701859790202535200",
                            "severityNumber": "SEVERITY_NUMBER_INFO",
                            "severityText": "Information",
                            "body": {
                                "stringValue": "Get Weatherforecast function called 2023-12-06 4:19:50 PM"
                            },
                            "attributes": [
                                {
                                    "key": "date",
                                    "value": {
                                        "stringValue": "2023-12-06 4:19:50 PM"
                                    }
                                },
                                {
                                    "key": "{OriginalFormat}",
                                    "value": {
                                        "stringValue": "Get Weatherforecast function called {date}"
                                    }
                                },
                                {
                                    "key": "SpanId",
                                    "value": {
                                        "stringValue": "d840418c7f429c48"
                                    }
                                },
                                {
                                    "key": "TraceId",
                                    "value": {
                                        "stringValue": "a4ba025a812e1d889c062a8319f7e951"
                                    }
                                },
                                {
                                    "key": "ParentId",
                                    "value": {
                                        "stringValue": "0000000000000000"
                                    }
                                },
                                {
                                    "key": "ConnectionId",
                                    "value": {
                                        "stringValue": "0HMVM93BK0GTU"
                                    }
                                },
                                {
                                    "key": "RequestId",
                                    "value": {
                                        "stringValue": "0HMVM93BK0GTU:00000003"
                                    }
                                },
                                {
                                    "key": "RequestPath",
                                    "value": {
                                        "stringValue": "/weatherforecast"
                                    }
                                },
                                {
                                    "key": "ActionId",
                                    "value": {
                                        "stringValue": "9eb2e807-14ab-410a-99a8-995b556f9362"
                                    }
                                },
                                {
                                    "key": "ActionName",
                                    "value": {
                                        "stringValue": "OtelLokiLogTest.Controllers.WeatherForecastController.Get (OtelLokiLogTest)"
                                    }
                                },
                                {
                                    "key": "name",
                                    "value": {
                                        "stringValue": "Eloise Stracke"
                                    }
                                },
                                {
                                    "key": "age",
                                    "value": {
                                        "intValue": "82"
                                    }
                                },
                                {
                                    "key": "address",
                                    "value": {
                                        "stringValue": "Klingport"
                                    }
                                }
                            ],
                            "traceId": "pLoCWoEuHYicBiqDGffpUQ==",
                            "spanId": "2EBBjH9CnEg=",
                            "observedTimeUnixNano": "1701859790202535200"
                        }
                    ]
                }
            ]
        }
    ]
}

Use Github Discussions.

[cijothomas]

non relevant attributes : LogRecord.attributes collection contains ActionId, ActionName, ConnectionId,

There are likely coming from ILogger scopes. The only option today to avoid these, is to not enable scopes support. (They are not enabled by default.)

There isn't any fine grained control over scopes today, and this is something that needs to be addressed.

[madhub]

non relevant attributes : LogRecord.attributes collection contains ActionId, ActionName, ConnectionId,

There are likely coming from ILogger scopes. The only option today to avoid these, is to not enable scopes support. (They are not enabled by default.)

There isn't any fine grained control over scopes today, and this is something that needs to be addressed.

Yes these are coming from log scope, but I need logger scope for application adds application specific attributes in logger scope. This is the only way to enrich logs with application specific attributes , need to have some extensibility to filter out the attributes that are not relevant for application .

List<KeyValuePair<string,object?>> logContext= new List<KeyValuePair<string,object?> 
{
    new KeyValuePair<string, object?>( "CustomerId","1234"),
    new KeyValuePair<string, object?>( "OrderId","768"),
}

using (logger.BeginScope( logContext ))
{
    logger.LogInformation("Processing credit card payment");
}

[madhub]

@cijothomas Can't we provide some option in OpenTelemetryLoggerOptions that takes Scope filter as an option to retain or remove items ? Or some way by installing custom export processor

simple pseudo code here,

public class OpenTelemetryLoggerOptions
{
      public Fun<bool ,...> ScopeFilter { get;set;}
}

[cijothomas]

Its definitely possible to offer some filtering ability, need to write up a proposal (or multiple), and gather feedback. Could you create a more specific issue to tackle this? ILogger is adding more filtering capability in the next release, and it may be worth checking if scope based filtering should come as part of that or it should be solved at OTel level. A separate issue will help track this issue.

[madhub]

Its definitely possible to offer some filtering ability, need to write up a proposal (or multiple), and gather feedback. Could you create a more specific issue to tackle this? ILogger is adding more filtering capability in the next release, and it may be worth checking if scope based filtering should come as part of that or it should be solved at OTel level. A separate issue will help track this issue.

Done #5239

[madhub]

Now I see more attributes in every log statement.Is there way to remove/disable it ?

"telemetry.sdk.language": "dotnet",
"telemetry.sdk.name": "opentelemetry",
"telemetry.sdk.version": "1.9.0"

[cijothomas]

Now I see more attributes in every log statement.Is there way to remove/disable it ?

"telemetry.sdk.language": "dotnet",
"telemetry.sdk.name": "opentelemetry",
"telemetry.sdk.version": "1.9.0"

There looks like Resources from OTel. No, they are not added to every statement. They are a separate concept, though it is entirely possible for a backend to copy all Resource attributes to every Log, but that is not done by OpenTelemetry/Exporters in this repo.

Doc for Resource: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/logs/customizing-the-sdk#setresourcebuilder

ResourceBuilder.CreateDefault() - is adding a bunch of spec-mandated defaults, including the ones like "telemetry.sdk.language" etc. If you don't want the defaults, you can use ResourceBuilder.CreateEmpty() and then add any resource you want explicitly. In other words, while OTel does have a default for Resource, it also allows one to override them to be empty.

[madhub]

"telemetry.sdk.language": "dotnet",
"telemetry.sdk.name": "opentelemetry",
"telemetry.sdk.version": "1.9.0"

Thanks . It is the ResourceBuilder CreateDefault() that added the TelemetrySdk details . Now with ResourceBuilder.CreateEmpty() and adding required service details , we don't see TelemetrySdk in log statements