Observability & Logs on ELK

[ijardillier]

We are using a complete Elastic stack to monitor our projects based on this IoT Edge LoraWan project.

It uses Prometheus to expose metrics and it is fully compatible with the Elastic stack (metricbeat module is able to read prometheus or openmetrics endpoints).

But for logs, is it possible to provide structured logs (as Json objetcs). To do this, it is possible for example to use NLog or Serilog loggers (just a library in top of Microsoft.Extensions.Logging, so it can use the ILogger interface as well).

Elastic provides Nugets to format log messages as Json objects and made them compliant with the (ECS).

For example, with Serilog, we can easily use the ECS formatter in the appsettings Serilog section :

"WriteTo": [ { "Name": "Console", "Args": { "formatter": "Elastic.CommonSchema.Serilog.EcsTextFormatter, Elastic.CommonSchema.Serilog" } } ]

and quite the same for NLog in the NLog section:

"extensions": [ { "assembly": "NLog.Extensions.Logging" }, { "assembly": "NLog.Web.AspNetCore" }, { "assembly": "NLog.Targets.ElasticSearch" }, { "assembly": "Elastic.CommonSchema.NLog" } ], "targets": { "console": { "type": "Console", "layout": { "type": "EcsLayout" } }

[Metal-Mighty]

Hi all! Adding to this enhancement by @ijardillier, if it gets implemented it would be very nice to also include context information to the published logs. One useful context information we would like would be the message identifier (CorrelationId ?) of the message received by the LNS from the LBS in order to track the message in the different application layers. This ID is already provided when forwarding messages to a different module with a route.

[bastbu]

Hello @ijardillier and @Metal-Mighty and thanks for your inputs! We were considering adding this feature, sadly it's not as straightforward as pulling in a NuGet package (yet). Since we are using Microsoft.Extensions.Logging and the generic host, we would need a different library (i.e. the Elasticsearch.Extensions.Logging part of ecs-dotnet) to support this, as we do not plan to depend on NLog or Serilog at the moment. Their repository has an open feature request, which we would like to reuse.

cc @Mandur

[Metal-Mighty]

Hi all, @bastbu I just noticed that the feature request you shared in your previous post has been closed (and apparently abandoned...) This topic is still important to us as it would allow us to better monitor and create alerts for our applications on our Elastic instances. Do you think it could still make its way to a future release ?

[Mandur]

Hello @metal-mighty,

We are not actively working toward this issue as it doesn't directly relate to LoRa/IoT support. From the last comment of the closed issue, I understand a console log formatter might still be in scope (I understand this is what you need). If such a formatter is implemented, we would naturally add the option to use it in the starterkit.

I would suggest you to follow up with the ecs-dotnet team on the issue to understand if there is plan for a formatter release in the future that we could include

[Metal-Mighty]

Hi @Mandur and thanks for your answer! I may have overlooked the last part of the message 😶 I'll try to follow up with them to see whether they can implement it.

In the mean time I added a new issue (#2069) to at least have a timestamp in console logs to help us during debug when we can't use our ELK logs.