Relationship with Prometheus.NET and Serilog

[dropsonic]

Question

What is the relationship between OpenTelemetry and a typical configuration with Prometheus.NET + Prometheus/Grafana for metrics and Serilog + ELK stack for logging, where Serilog is configured to be used with Microsoft.Extensions.Logging? Should I use one over the other or in some combination?

[ElectricVampire]

I am also looking for answer of above question. I am using Serilog for logging - I enrich all the logs with TraceID and SpanID of Http Request. I am using Prometheus.NET to collect metrics. How do I leverage OpenTelemetry.NET in this setup?

I also have one more problem- I receive messages from Third Party MF system to my kafka queue. Each message has its own unique id. There is .Net core kafka message processor which reads message from Queue and call a multiple Web Apis. How do I assign unique Id of message to System.Diagnostics.Activity.TraceID in Kafak .Net processor, so any HttpClient call made from this exe to any API get that copied over via request headers.

Alternatively please also let me know if I am approaching the problem in wrong way and there is already a standard solution for this kind of problem.

[clarkezone]

I'm interested in the related question of how i can use opentelemetry.net with Loki for logs

[RehanSaeed]

Once the Open Telemetry Logging is fully baked, then I think there will be no need to use Serilog anymore. Instead, you'd simply configure an Open Telemetry exporter for your Traces, Metrics and Logs (You have to do each one separately in Open Telemetry .NET).

Open questions for me are whether Jaeger, ElasticSearch or others support importing logs or even metrics using the OpenTelemetry Protocol (OTPL). My understanding is that they both support traces at least.

In the mean time, you can continue to use Serilog with my Serilog.Enrichers.Span package which will connect your logs to your traces by embedding Span ID's into your logs.

[damikun]

@RehanSaeed What i know is that opentelemetry dont wanna provide custom logging solution just adapter to existing.. So Serilog will keep doing what it does just provide sink to export logs over otel protocol or opentelemetry provides custom listener for that..

So just to explain openetlemetry will not cover logging itself across languages just provides adapter to listen different logging solutions over different languages..

Otel Collector already supports logging input and you can export that to multiple endpoints as one of it is elastic APM.. What is currently missing (in time writing this) is support of exporting logs from client libraries (the adapters) but you can build it by yourself quite fast until all SDK adds support...

https://github.com/damikun/trouble-training/blob/main/Doc/OpenTelemetry.md#logs-vs-telemtry

[queil]

@clarkezone As for Loki, OpenTelemetry.NET, and Serilog (unless you've already figured it out). I think the desired solution would be to have an OpenTelemetry-Collector endpoint that Serilog pushes logs to via a compatible serilog sink (from what I could see there is no open-source package that would do that yet). Then you can have a Loki exporter configured in your OpenTelemetry-Collector. My current POC solution involves enriching the logs with Serilog.Enrichers.Span so they contain trace ID and pushing directly to Loki via Serilog.Sinks.Grafana.Loki. Then you could get something this blogpost describes.

[cijothomas]

relationship between OpenTelemetry and a typical configuration with Prometheus.NET + Prometheus/Grafana

OpenTelemetry Metrics API (for .NET, it is packed into System.Diagnostics.DiagnosticSource package from runtime itself), is vendor/backend neutral. You may export to Prometheus or any other backends, without changing instrumentation code.

On the other hand, using Prometheus .NET, ties you to Prometheus backend.

You can still use Grafana in either approach, but OpenTelemetry approach is more neutral, and hence we (obviously!) recommend this route :)

[cijothomas]

1. One of the OpenTelemetry's mission to provide vendor/backend agnostics instumentation API. So if you use OTel API - then you can switch backends without changing instrumentation side.
2. OpenTelemetry is still evolving, so may not have all the features as other existing solutions have.
3. If there are more questions about OpenTelemetry, please open specific issues, to get guidance from others.

Closing this issue, as the original question has been answered. Please consider opening a new discussion thread, for more clarifications, if needed.

[liammclennan]

In case anyone else finds this thread, please note that Serilog does have an OpenTelemetry sink that can send logs to any OTLP logging backend.

[liammclennan]

There is also work underway to add hierarchical, distributed tracing to Serilog.