search

serilog-contrib / serilog-sinks-elasticsearch

A Serilog sink that writes events to Elasticsearch
Apache License 2.0
434 stars 197 forks source link

Support for Elastic Common Schema #227

Open rezvanf opened 5 years ago

rezvanf commented 5 years ago

A few questions before you begin:

Is this an issue related to the Serilog core project or one of the sinks or community projects.
This issue list is intended for Serilog Elasticsearch Sink issues. If this issue relates to another sink or to the code project, please log on the related repository. Please use Gitter chat and Stack Overflow for discussions and questons.

Does this issue relate to a new feature or an existing bug?

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package. latest nuget

What is the target framework and operating system? See target frameworks & net standard matrix.

Please describe the current behavior? n/a

Please describe the expected behavior? Introduce fluent api to enable ecs compliant fields to be logged

Introduce a json sink that writes out compliant ecs json straight to elastic search.

I have already started an implementation of this in my organisation and would be happy to add this functionality through a PR.

If the current behavior is a bug, please provide the steps to reproduce the issue and if possible a minimal demo of the problem

mivano commented 5 years ago

I had not heard about Elastic Common Schema, but it sounds very useful! Would certainly welcome a PR as long as it does not break older clients and can be switched on or off. Curious to see what you come up with!

rezvanf commented 5 years ago

Great. Will do.

This is ecs btw.

https://www.elastic.co/blog/introducing-the-elastic-common-schema

mivano commented 5 years ago

More reading to do this weekend :-) Nice move to a standard which will allow easier dashboarding and queries, I like it.

mivano commented 5 years ago

@rezvanf any progress on this ECS? Are you blocked somewhere?

rezvanf commented 5 years ago

So I have a 1st draft i am just working with my organisation on how i can contribute back without breaking my organisations policies.

spfaeffli commented 5 years ago

@mivano, @rezvanf any news on this?

rezvanf commented 5 years ago

Yes I am working on to put a PR through. I have a version of the code which supports a json formatter with enrichers, which support some of the ECS fields. The output i have had reviewed directly with Elastic themselves. This has taken me some time because the code is coming through the organisation i work for and therefore I need to consider the IP implications. Can you please help and tell me where the contributing guide is ..I would like to know which branch i need to put my PR to.

rezvanf commented 5 years ago

I have just noticed it is the dev branch that the PR should go into.

The current elastic search sinks targets .net 1.1 is this something we wish to keep?

As 1.1 will reach of end of support in 20 days.

mivano commented 5 years ago

Sorry, yes use dev as a start branch. Then I can create test packages. It also now contains an updated version that removed 1.1 support.

lancegliser commented 5 years ago

Hi @rezvanf, I'm coming up against this problem myself. Have any progress or base pieces we could work from?

rezvanf commented 5 years ago

I am attempting to build my projects that I have added to the existing solution. My projects target 2.2 and I am getting this error Severity Code Description Project File Line Suppression State

Error NETSDK1045 The current .NET SDK does not support targeting .NET Core 2.2. Either target .NET Core 2.1 or lower, or use a version of the .NET SDK that supports .NET Core 2.2. Serilog.Formatting.Elasticmiddleware C:\Program Files\dotnet\sdk\2.1.505\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.TargetFrameworkInference.targets 137

rezvanf commented 5 years ago

This is the installed SDK .NET Core SDKs installed: 2.1.403 [C:\Program Files\dotnet\sdk] 2.1.502 [C:\Program Files\dotnet\sdk] 2.1.505 [C:\Program Files\dotnet\sdk] 2.1.602 [C:\Program Files\dotnet\sdk] 2.2.100 [C:\Program Files\dotnet\sdk]

ledouxpl commented 5 years ago

@rezvanf @mivano Any update on the PR? I would gladly contribute if a feature branch is made public.

mivano commented 5 years ago

Sorry, I do not have any more information. @Mpdreamz are you also still interested in this development?

Mpdreamz commented 5 years ago

Hi! Yes we will be opening up a repository in the coming week that will aid with this effort.

We will take a more active and more importantly open approach following that, anyone's peering eyes and contributions will be very much appreciated.

cc @codebrain

rezvanf commented 5 years ago

i have a working solution, i just have the challenge of getting from my place of work into the outside world here on github. It has to go through IP checks, which is time consuming unfortunately.

mhoyer commented 5 years ago

Just for the sake of completeness, these are the links to the ESC field references:

Maybe we can generate the C# sources from that schema.json?

In addition I found this repo: https://github.com/ehsan-maleki/serilog-sinks-elastic-ecs-logstash

Mpdreamz commented 4 years ago

Hi everyone,

We've opened our repos that we have for ECS to be used with .NET

https://github.com/elastic/ecs-dotnet

It includes a generator that can emit C# sharp types from the ECS schema.

The types can serialize/deserialize by themselves to an ecs json log format. We are still debating what that means exactly but for now it makes sure certain properties are always first (timestamp, log.level, message) so that if persisted to file it remains somewhat scannable.

The repos also adds a Serilog TextFormatter that can be used with many different sinks. We try to play nice with Serilog.Enrichers.* packages OOTB.

We expect to release initial beta's in the coming weeks.

mivano commented 4 years ago

Great progress! The TextFormatter is a nice solution. Thanks for the update.

orjan commented 4 years ago

@Mpdreamz this formatter is really awesome and together with data streams we're having a sensible default almost out for the box! Is there anything more we'll need to do in this sink more that setting the Elastic.CommonSchema.Serilog.EcsTextFormatter to fully support it?

@mivano I didn't know about this formatter, what about promoting it a little bit more in the readme?

orjan commented 4 years ago
Screenshot 2020-09-24 at 21 39 38

I'm not sure if this is related to this sink, the formatter or something lacking in Kibana? But being able to filter on values from serilog is important.

References

mivano commented 4 years ago

Totally in favor of promoting this more, however, my knowledge is pretty limited. But it looks really interested and a nice way forward for the ES sink.