[BUG] Not all Properties are Logged as Values in ES like "ElapsedMilliseconds"

[squadwuschel]

ECS integration/library project(s) (e.g. Elastic.CommonSchema.Serilog): "Elastic.Apm.SerilogEnricher" Version="8.11.0" "Elastic.Serilog.Sinks" Version="8.11.0"

Elasticsearch version (if applicable): 8.8.x

.NET framework / OS: .net core 7

Description of the problem, including expected versus actual behavior:

I try to log the following logmessage:

_logger.LogInformation("Response time for {Route}: {ElapsedMilliseconds}ms for {TraceId}", requestRoute, stopwatch.ElapsedMilliseconds, traceId);

when I search for this log entry in Kibana I found the Message but only the "Route" is stored into an extra field. The "ElapsedMilliseconds" and "TraceId" I can't find as extra values which I can use in my Kibana dashboards. To be clear, I can't find these fields in the stored Json Document in ES only in the MessageTemplate I can see that the Values are "set"

message: Response time for "/v1/weatherforecast": 237ms for "0HN3QD9B6FVH5:0000000F"

When I Rename "ElapsedMilliseconds" to "Milliseconds" then I get an extra field I can use in my dashboards.

Are there some reserved Variable Names I am not able to use or is this a bug?

thats my confoiguration from ES and Serilog

     loggerConf.Enrich.FromLogContext()
            .Enrich.FromLogContext()
            .Enrich.WithMachineName()
            .Enrich.WithExceptionDetails()
            .Enrich.WithElasticApmCorrelationInfo()
            .Enrich.WithProperty("Environment", hostContext.HostingEnvironment.EnvironmentName)
            .Enrich.WithProperty("ApplicationName", applicationName)
            .Enrich.With(enricher.ToArray())
            .WriteTo.Debug()
            .WriteTo.Console()
            .WriteTo.Elasticsearch(
                new[] { elasticSearchAccountConfiguration!.ElasticsearchUrl },
                opts =>
                {
                    opts.DataStream = new DataStreamName(LoggerPrefix, applicationName, hostContext.HostingEnvironment.EnvironmentName);
                    opts.BootstrapMethod = BootstrapMethod.None;

                },
                transport =>
                {
                    transport.Authentication(
                        new BasicAuthentication(
                            elasticSearchAccountConfiguration.Username,
                            elasticSearchAccountConfiguration.Password
                        )
                    );
                }
            )

[andreycha]

Hi @squadwuschel , The library by default detects properties with names Elapsed and ElapsedMilliseconds whose values can be parsed as doubles and does not keep them as properties, but maps directly to event.duration field in ECS instead. See: https://github.com/elastic/ecs-dotnet/blob/8136f2ccfd90793fa3d6cff04faea1af64d86f1e/src/Elastic.CommonSchema.Serilog/LogEventConverter.cs#L250-L253 And: https://github.com/elastic/ecs-dotnet/blob/8136f2ccfd90793fa3d6cff04faea1af64d86f1e/src/Elastic.CommonSchema.Serilog/LogEventConverter.cs#L160-L161

[squadwuschel]

Hi @andreycha,

thnaks that was a very fast answer and good to know.

Hint: The field "event.duration" is stored as nanoseconds

And what about the "TraceId" why can't I find this value in the Logs?

[squadwuschel]

It seems that this is parsed into the field:

"trace.id": "0HN3QD9B6FVH5:0000000F"

[andreycha]

Yes. It's another feature of the library, ECS aware properties, which is mentioned in the docs for every log framework, e.g. https://github.com/elastic/ecs-dotnet/blob/8136f2ccfd90793fa3d6cff04faea1af64d86f1e/src/Elastic.CommonSchema.Serilog/README.md#ecs-aware-message-templates.

The library recognizes the properties which match standard ECS fields and write values directly into standard fields. You can find the full list of properties here: https://github.com/elastic/ecs-dotnet/blob/8136f2ccfd90793fa3d6cff04faea1af64d86f1e/src/Elastic.CommonSchema/LogTemplateProperties.Generated.cs#L3281