serilog-contrib / serilog-sinks-elasticsearch

A Serilog sink that writes events to Elasticsearch
Apache License 2.0
434 stars 197 forks source link

Integrate `Elasticsearch.CommonSchema.Serilog` text formatter #494

Open nenadvicentic opened 1 year ago

nenadvicentic commented 1 year ago

A few questions before you begin:

Is this an issue related to the Serilog core project or one of the sinks or community projects.
This issue list is intended for Serilog Elasticsearch Sink issues. If this issue relates to another sink or to the code project, please log on the related repository. Please use Gitter chat and Stack Overflow for discussions and questions.

Does this issue relate to a new feature or an existing bug?

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package. 9.0.0-alpha-*

What is the target framework and operating system? See target frameworks & net standard matrix.

As a part of integration effort between .NET and Elasticsearch, Elastic team provides Elastic Common Schema .NET repository which includes two sub-projects:

The issue is a follow-up to #254 and #227, containing more up-to-date information and first look at the integration.

Simple experiment with naïve Serilog.Sinks.Elasticsearch integration, where EcsTextFormatter simply replaces current default formater ElasticsearchJsonFormatter:

public static ITextFormatter CreateDefaultFormatter(ElasticsearchSinkOptions options)
{
    //return new ElasticsearchJsonFormatter(
    //    formatProvider: options.FormatProvider,
    //    closingDelimiter: string.Empty,
    //    serializer: options.Serializer != null ? new SerializerAdapter(options.Serializer) : null,
    //    inlineFields: options.InlineFields,
    //    formatStackTraceAsArray: options.FormatStackTraceAsArray
    //);

    return new EcsTextFormatter();
}

...produces following output in Elasticsearch/Kibana v8.6.0:

image

Open questions would be:

nenadvicentic commented 1 year ago

Currently, using Serilog.Sinks.Elasticsearch v9.0.0, it is possible to integrate Elastic.CommonSchema.Serilog.EcsTextFormatter using the customFormatter options, as shown in the extract of appsettings.json bellow:

{
  // ....
  "Serilog": {
    "Using": [ "Serilog.Sinks.Elasticsearch" ],
    "MinimumLevel": "Information",
    "WriteTo": [
      {
        "Name": "Elasticsearch",
        "Args": {
          "nodeUris": "http://localhost:9200",
          "customFormatter": "Elastic.CommonSchema.Serilog.EcsTextFormatter, Elastic.CommonSchema.Serilog"
        }
      }
    ],
    "Enrich": [ "FromLogContext", "WithMachineName" ],
    "Properties": {
      "Application": "My app"
    }
  },
  // ...
}

However, it turned out that current version of Elastic.CommonSchema.Serilog.EcsTextFormatter from Elastic.CommonSchema.Serilog NuGet package v1.5.3 only works with Elasticsearch server v8.x! It breaks logging when pointed to Elasticsearch server versions lower than 8. With it's default formatter Serilog.Sinks.Elasticsearch v9.0.0 works against Elasticsearch server major versions 6, 7 and 8.