search

elastic / ecs-dotnet

https://www.elastic.co/guide/en/ecs-logging/dotnet/current/setup.html
Apache License 2.0
119 stars 62 forks source link

Release new version of ECS.NET 8.6.0 #197

Closed Mpdreamz closed 1 year ago

Mpdreamz commented 2 years ago
bonyaroslav commented 2 years ago

@Mpdreamz @gregkalapos Any news when we can use it? We're waiting for many months already

odin568 commented 2 years ago

Same for me. Eager to use updated version to get rid of some workarounds

lewinskimaciej commented 1 year ago

Is this package still being maintained? We desperately need https://github.com/elastic/ecs-dotnet/issues/167 but it seems that there hasn't been a release in 1,5 years. Is there something we can do to help push this out?

Mpdreamz commented 1 year ago

Still maintained folks! Thanks for the nudges for an update.

I have been on leave for an extended period of time but am back working on getting an update out the door. Any feedback on the upcoming pre-release would be most welcomed!

Will comment on this issue when it goes up on nuget.

Mpdreamz commented 1 year ago

Heads up here that we just shipped version 8.4.0-alpha1 to nuget.

I intend to wait two weeks to collect feedback on this new version before shipping it as a GA release (8.4.0).

A LOT of changes have gone in since we last release 1.6.0-alpha1, you can see the full list of changes here

Here are some highlights:

New integrations

Big Updates:

Breaking Changes.

HubDevUser commented 1 year ago

Can you please include .netstandard2.0 in the official 8.4.0 release. We need to consume this in fullframework, the alpha1 release only contains .netstandard2.1 assemblies @Mpdreamz

reydelleon-skuvault commented 1 year ago

@Mpdreamz Are there plans to update documentation to show how to put all these together? For example, how to enhance the EcsDocument with custom fields.

I noticed that you're putting forward an alternative to Serilog.Sinks.Elasticsearch. it would be helpful to see some examples of how you would configure your sink to achieve a similar experience (knowing that they don't have the same features) than that of the contrib sink. Many of us have been using that other sink for a long time and would like to make any possible migration as painless as possible.

Mpdreamz commented 1 year ago

@reydelleon-skuvault absolutely, docs are a big prerequisite for the 8.4.0 release.

I tried to capture some of the main differences between the sinks here for now: https://github.com/elastic/ecs-dotnet/tree/main/src/Elastic.CommonSchema.Serilog.Sink#comparison-with-serilogsinkselasticsearch

Any feedback or additional questions you may have are more than welcomed!

Mpdreamz commented 1 year ago

A quick extremely raw example of custom EcsDocument implementation is here: https://github.com/elastic/ecs-dotnet/blob/main/src/Elastic.CommonSchema.BenchmarkDotNetExporter/Domain/BenchmarkDocument.cs#L16

HubDevUser commented 1 year ago

How can i set a custom index, it always defaults to ecs-dotnet-logs ?

Mpdreamz commented 1 year ago

How can i set a custom index, it always defaults to ecs-dotnet-logs ?

Using what integration?

HubDevUser commented 1 year ago

Using aspnet core similar to the example you added earlier examples/aspnetcore-with-extensions-logging

Mpdreamz commented 1 year ago

Using aspnet core similar to the example you added earlier examples/aspnetcore-with-extensions-logging

If you are using Elasticsearch.Extensions.Logging you can configure it in appsettings.json

{
  "Logging": {
    "Elasticsearch": {
      "ShipTo": {
        "NodePoolType": "Static",
        "NodeUris": [ "http://localhost:9200" ]
      },
      "DataStream": {
        "DataSet": "my.application"
      }
    },
    "LogLevel" : {
      "Default": "Trace",
      "Microsoft": "Warning"
    }
  }
}

This will log to logs-my.application-default.

You can also configure this in the application builder.

builder.Host.ConfigureLogging((_, loggingBuilder) =>
{
    loggingBuilder.AddElasticsearch(opts =>
    {
        opts.DataStream = new DataStreamNameOptions
        {
            Type = "logs", DataSet = "dotnet", Namespace = "default"
        }
    }),
Mpdreamz commented 1 year ago

Can you please include .netstandard2.0 in the official 8.4.0 release. We need to consume this in fullframework, the alpha1 release only contains .netstandard2.1 assemblies @Mpdreamz

I've just pushed 8.4.0-alpha2 with netstandard2.0 support.

HubDevUser commented 1 year ago

It doesn't contain a .netstandard2.0 under libs, can you take a look please ?

reydelleon-skuvault commented 1 year ago

@Mpdreamz I'm trying to use the new sink but I'm not seeing a way to specify auth credentials when configuring it. There is nothing about this in the README and looking through the code I wasn't able to identify a way to do this.

Can you provide some guidance on this and perhaps add something in the examples in the README?

lewinskimaciej commented 1 year ago

@Mpdreamz I too can't seem to find a way of setting auth on new Serilog Elastic sink. I think this would be achievable by exposing setter for ElasticsearchSinkOptions.Transport or some other, maybe nicer way. Currently it's always DefaultHttpTransport with default TransportConfiguration.

HubDevUser commented 1 year ago

@reydelleon-skuvault @lewinskimaciej see https://github.com/elastic/ecs-dotnet/pull/267#discussion_r1107859406 for example on setting up the auth options

HubDevUser commented 1 year ago

@Mpdreamz how can we configure for .netfullframework where ILogger is not an option?

LiorBanai commented 1 year ago

@HubDevUser you can use ILogger in net framework using the nuget:

        <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="7.0.0" />
reydelleon-skuvault commented 1 year ago

@reydelleon-skuvault @lewinskimaciej see https://github.com/elastic/ecs-dotnet/pull/267#discussion_r1107859406 for example on setting up the auth options

@HubDevUser The example you linked is for the logging provider, not the sink. @Mpdreamz We need an example that works for the sink. Do we have one?

reydelleon-skuvault commented 1 year ago

Found the way to do it, but it is rather convoluted if all I want to change is the authentication. 

var transportConfig = new TransportConfiguration(new Uri("http://localhost:9200/"));
transportConfig.Authentication(new BasicAuthentication("elastic", "some-password"));
var transport = new DefaultHttpTransport(transportConfig);
var sinkOptions = new ElasticsearchSinkOptions(transport)
{
    ...
};
loggerConfiguration
    .WriteTo.Elasticsearch(sinkOptions);

@Mpdreamz Is there a better way to do this?

Mpdreamz commented 1 year ago

@reydelleon-skuvault thanks for bringing this to my attention I've opened https://github.com/elastic/ecs-dotnet/pull/286 to address this usability issue.

With that you should be able to use:

.WriteTo.Elasticsearch(nodes, opts =>
{
    opts.BootstrapMethod = BootstrapMethod.Failure;
    opts.DataStream = new DataStreamName("logs", "console-example");

}, transport =>
{
    transport.Authentication();
})
Mpdreamz commented 1 year ago

I just pushed 8.4.0-alpha3 to nuget that includes netstandard2.0 for all projects and includes the fix to make it easier to configure the transport options when using the serilog sink.

Thanks everyone for kicking the tires!

Mpdreamz commented 1 year ago

I just pushed ECS.NET 8.4.0-alpha4 that includes everyones feedback and PR's much obliged to everyone kicking the tires!

I opened #291 to bump us to 8.6.0 and this includes some non breaking changes to how we bootstrap index templates. This PR will allow new version to upgrade template indices.

reydelleon-skuvault commented 1 year ago

@Mpdreamz Is there a good way to see logs about the sink activity? I have SelfLog enabled for Serilog and the sink configured but thought I'm not getting any data in Elasticsearch, there is no indication as to what is the cause for this. Nothing written to the self log file (is it supposed to?).

I have disabled pinging (a shot in the dark) but that didn't resolve it.

I do have logs in the Console and local file, so there is content. It is just not making it to Elasticsearch.

Mpdreamz commented 1 year ago

We are logging all export errors to Serilog's Self log:

https://github.com/elastic/ecs-dotnet/blob/af53e3cdb8d00858b62f6d071b442163a9bfad9f/src/Elastic.CommonSchema.Serilog.Sink/ElasticsearchSink.cs#L77

The following diagnostics method is still ugly and not intended to make it to 1.0 but we ship with something called a ChannelListener<> if you do the following. 

    .WriteTo.Elasticsearch(....
    {
        ConfigureChannel = channelOpts =>  {
            channelOpts.BufferOptions = new BufferOptions { ExportMaxConcurrency = 10 };

            SomeStaticPlace = new ChannelListener<EcsDocument, BulkResponse>().Register(channelOpts);
        }
    })

What does SomeStaticPlace.ToString() report?

reydelleon-skuvault commented 1 year ago

@Mpdreamz This is what I get from the code you posted:

Failed publish over channel: ChannelListener.
Exported Buffers: 0
Exported Items: 0
Export Responses: 0
Export Retries: 0
Export Exhausts: 0
Inbound Buffer Read Loop Started: False
Inbound Buffer Publishes: 0
Inbound Buffer Publish Failures: 0
Outbound Buffer Read Loop Started: False
Outbound Buffer Read Loop Exited: False
Outbound Buffer Publishes: 0
Outbound Buffer Publish Failures: 0

Exception:

That's it. No more details than that. Here is my setup (without the diagnostic code), in case I'm doing something wrong here:

loggerConfiguration
            .WriteTo.Elasticsearch<EnhancedEcsDocument>(
                new[] {new Uri("http://localhost:9200")},
                options =>
                {
                    options.DataStream = new DataStreamName("logs", "myservice", environmentName.ToLowerInvariant());
                    options.BootstrapMethod = BootstrapMethod.None ;
                    };
                },
                transportConfig =>
                    transportConfig
                        .Authentication(new BasicAuthentication( "someUSer", "somePassword"))
                        .DisablePing()
            );
reydelleon-skuvault commented 1 year ago

I ended up (successfully) shipping the logs to a different Elasticsearch server, which means two things: 1. the problem was the server I was trying to ship to and 2. The sink works as expected.

That said, I still don't know what the problem with the original server was because I didn't get anything in the Self log.

Mpdreamz commented 1 year ago

@reydelleon-skuvault Thanks for reporting back another instance works!

I would still love to improve the experience here and understand why the other instance does not work.

Can you run the following in a new console application? Of course using 

ChannelListener<EnhancedEcsDocument, BulkResponse>? listener = null;
var waitHandle = new CountdownEvent(1);

// -- Setup Serilog --
var nodes = new[] { new Uri("http://localhost:9200") };
Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Debug()
    .MinimumLevel.Override("Microsoft", LogEventLevel.Information)
    .Enrich.FromLogContext()
    .WriteTo.Elasticsearch<EnhancedEcsDocument>(nodes, opts =>
    {
        opts.BootstrapMethod = BootstrapMethod.None;
        opts.DataStream = new DataStreamName("logs", "console-example");
        opts.ConfigureChannel = channelOpts => {
            channelOpts.BufferOptions = new BufferOptions
            {
                ExportMaxConcurrency = 1,
                OutboundBufferMaxSize = 2,
                WaitHandle = waitHandle
            };
            listener = new ChannelListener<EnhancedEcsDocument, BulkResponse>().Register(channelOpts);
        };
    }, 
    transportConfig =>
        transportConfig
            .Authentication(new BasicAuthentication( "someUSer", "somePassword"))
            .DisablePing()
    )
    .CreateLogger();

// -- Log 2 items and wait for flush --
Log.Logger.Information("Writing event 1");
Log.Logger.Information("Writing event 2");

if (!waitHandle.WaitHandle.WaitOne(TimeSpan.FromSeconds(10)))
    throw new Exception($"No flush occurred in 10 seconds: {listener}", listener?.ObservedException);
else
{
    Console.WriteLine("Successfully indexed data to Elasticsearch");
    Console.WriteLine(listener);
}

Thanks in advance!

reydelleon-skuvault commented 1 year ago

@Mpdreamz This is what I get:

Successfully indexed data to Elasticsearch
Failed publish over channel: ChannelListener.
Exported Buffers: 1
Exported Items: 2
Export Responses: 1
Export Retries: 0
Export Exhausts: 0
Inbound Buffer Read Loop Started: True
Inbound Buffer Publishes: 2
Inbound Buffer Publish Failures: 0
Outbound Buffer Read Loop Started: True
Outbound Buffer Read Loop Exited: False
Outbound Buffer Publishes: 1
Outbound Buffer Publish Failures: 0

Exception: System.Exception: Unsuccessful () low level call on POST: /_bulk?filter_path=error%2C%20items.%2A.status%2Citems.%2A.error
 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendi
ngRequestsCts, CancellationToken originalCancellationToken)
   at Elastic.Transport.HttpTransportClient.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---

Process finished with exit code 0.

Once more, I think the issue has to do with my server setup in this case. There seems to be more info here than in my previous attempt. Note that although there are no publishing failures reported in the metrics, the logs in this case never made it to Elasticsearch. For comparison, here is the output using a different server, where the credentials are wrong.

Exception: System.Exception: Unsuccessful (401) low level call on POST: /_bulk?filter_path=error%2C%20items.%2A.status%2Citems.%2A.error
 ---> Elastic.Transport.TransportException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration. Call: Status code 401 from: POST /_bulk?fil
ter_path=error%2C%20items.%2A.status%2Citems.%2A.error
 ---> Elastic.Transport.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.
   at Elastic.Transport.DefaultRequestPipeline`1.ThrowBadAuthPipelineExceptionWhenNeeded(ApiCallDetails details, TransportResponse response)
   at Elastic.Transport.DefaultRequestPipeline`1.CallProductEndpointAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elastic.Transport.DefaultHttpTransport`1.RequestAsync[TResponse](HttpMethod method, String path, PostData data, RequestParameters requestParameters, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---

[UPDATE] This is what I found when digging on the server side logs:

2023-04-03T16:46:23.723704200Z {"@timestamp":"2023-04-03T16:46:23.723Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=redacted, remoteAddress=redacted}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#10]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"2cKEp0AMR1GWWNSxsG4mmQ","elasticsearch.node.id":"fDnS7aKgRmWtqYHMLs6TZA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"elastic-cluster"}

This seems to explain the exception outputted by the listener. There is still the fact that I'm not seeing any of this in the self-log, but that might also be an issue with my own setup.

reydelleon-skuvault commented 1 year ago

@Mpdreamz Is there documentation around SSL certificate configuration. What would be the best course of action if we're using a test Elasticsearch server (in Docker container, for example) that has a self signed certificate? Can I disable certificate validation on the sink for testing?

reydelleon-skuvault commented 1 year ago

@Mpdreamz I got another question about customizing the EcsDocument.

The examples I have found peppered in READMEs and tests all show contrived examples of how to subclass EcsDocument to introduce custom properties. They seem to suggest that you can do nesting (think the Contoso class with CompanyName as a nested property). In the TryRead, ReceiveProperty and WriteAdditionalProperties methods the examples show us handling the root property. Take this test for example: https://github.com/elastic/ecs-dotnet/blob/main/tests/Elastic.CommonSchema.Serilog.Tests/Repro/GithubIssue167.cs

Here is the Tryread() method for visibility:

protected override bool TryRead(string propertyName, out Type type)
{
    type = propertyName switch
    {
        "contoso" => typeof(Contoso),
        _ => null
    };
    return type != null;
}

The problem that I see with this is that I don't necessarily set the whole object in a LogEvent property. I might enrich the LogEvent with the contoso.company_name and set it to a single string value, but I will not set a contoso property in the LogEvent and set the value to the serialized contents of the object.

I'm confused as to how all this should work (adding custom properties to EcsDocument). The benchmark example doesn't use Serilog. Is it that this only works when using the Elastic agent directly, without Serilog? Does it work with Serilog at all? Maybe I have to use the MapCustom function when working with Serilog and do the mapping myself?

Perhaps what I'm missing is how things should look like in the Serilog LogEvent for this to work.

Any guidance on this would be appreciated, since I need to add properties (not in metadata/labels).

Mpdreamz commented 1 year ago

@Mpdreamz Is there documentation around SSL certificate configuration. What would be the best course of action if we're using a test Elasticsearch server (in Docker container, for example) that has a self signed certificate? Can I disable certificate validation on the sink for testing?

.WriteTo.Elasticsearch(nodes, opts =>
{
    //
}, transport =>
{
    transport.ServerCertificateValidationCallback(CertificateValidations.AllowAll);
})
Mpdreamz commented 1 year ago

Just a heads up that today we (finally) released a GA version of the ECS .NET libraries

New documentation: https://www.elastic.co/guide/en/ecs-logging/dotnet/current/intro.html

You should be able to upgrade to 8.6.0 today!

Thanks a ton for everyone involved, all the contributions, feedback and questions truly helped make this a mammoth of a release. It took a considerable time and effort to move us from 1.6.0-alpha1 to 8.6.0 with the main issue being specification format changes requiring us to retool our code generation from the ground up.

Formatters:

Datashippers:

Enrichers:

While all ECS logging libraries are Activity aware for distributed tracing, installing the following enrichers will use even richer tracing information from the Elastic APM Agent https://github.com/elastic/apm-agent-dotnet

New architecture:

We laid out a completely new architecture for our datashippers in this repository now taking advantage of https://github.com/elastic/elastic-ingest-dotnet System.Threading.Channel backed abstractions to push buffered data to any datasource.

image

Is a dedicated ECS event data shipper that can bootstrap target datastreams/indices with the appropriate ECS component templates and settings.