Support one or more of OpenTracing, OpenCensus, OpenTelemetry technologies

[SergeyKleyman]

The request to support OpenTracing was mentioned and now after OpenTracing and OpenCensus merged into OpenTelemetry maybe we should shoot directly OpenTelemetry?

It's indirectly related to elastic/apm-agent-dotnet#229.

[gregkalapos]

I think we should go with OpenTelemetry. OpenTelemetry plans to be backwards compatible with OpenCensus and OpenTracing (it'd be some bridge or something like that, I don't know the details, but this is definitely on their roadmap). Also as it seems in case we'd start working on either OpenCensus or OpenTracing, those may be already deprecated when we release.

So with the future of those 3 projects plus the plan OpenTelemetry offering support for the 2 others ones I'd vote for OpenTelemetry.

[SergeyKleyman]

To expand a little bit on

It's indirectly related to #229

It seems that the main motivation for the users to use one of OpenTracing, OpenCensus, OpenTelemetry API instead of Elastic .NET agent public API directly is to keep application code APM vendor neutral. DiagnosticSource API achieves this goal as well and from our experience it seems better adopted by .NET ecosystem than any of OpenTracing, OpenCensus, OpenTelemetry because DiagnosticSource is backed by Microsoft. So it is reasonable to conclude that starting with #229 DiagnosticSource will reach more users and allow them to keep application code APM vendor neutral.

[yhvicey]

https://medium.com/opentracing/a-roadmap-to-convergence-b074e5815289 looks like OpenTracing and OpenCensus plans to merge together.

[justinaspeciura]

I understand you want to maximize reach by waiting more broad APIs but OpenTracing is de-facto standard at this moment and no-one knows how long it will take to for them to merge or provide any packages/APIs.

You already provide OpenTracing bridges in most other major languages, why not C# too?

[gregkalapos]

You already provide OpenTracing bridges in most other major languages, why not C# too?

This has some practical reasons - .NET is the newest agent, and by the time we started with .NET the OpenTelemetry project was started and we already knew that OpenTracing won't be continued.

[megakid]

I can understand your reasoning but I think it's a mistake to not put the effort into building an OpenTracing bridge for .NET.

We conducted an in-depth look into distributed tracing options for our entire .NET code base approximately 3 months ago and for a variety of reasons, OpenTracing is still the number 1 provider-agnostic tracing instrumentation lib that supports both .NET Core + .NET Framework nicely. We saw that OpenTelemetry is coming but they themselves have promised full compatibility/bridging with OpenTracing and (at the time) said to use OpenTracing if you want to introduce tracing to your code (unsure what their advice is now but the OpenTelemetry .NET client is probably a year away from production-ready). We aim to use DiagnosticSource/Activity native .NET Core APIs in time but we currently still have a bunch of .NET Framework needs which OpenTracing instrumentation satisfies nicely.

As raised with our elastic account manager last week, this is currently a blocker for us moving to Elastic APM and we have just started rolling out Jaeger (backed with elastic so we still use elastic in some form) to satisfy our tracing requirements. Given your existing Java bridging library and the similarities between OpenTracing instrumentation on Java vs C#, shouldn't this relatively easy to achieve?

[gregkalapos]

Thanks for the input @megakid,

but the OpenTelemetry .NET client is probably a year away from production-ready

I wanna be careful what statement I make about the OpenTelemetry .NET project, but I tend to agree with what you say. This is the main reasons why we did not work on OpenTelemetry support yet for .NET.

Given your existing Java bridging library and the similarities between OpenTracing instrumentation on Java vs C#, shouldn't this relatively easy to achieve?

We can take all the learnings that we had with Java regarding turning OpenTracing format to our Elastic APM format, so that's good. On the other hand the code needs to be rewritten here in this repo, so there is no way to share code I think, this would be a new implementation of the bridge.

Do you have any experience with OpenTracing backwards compatibility of OpenTelemetry? A couple of month ago I prototyped a C# OpenTelemetry exporter that sends OpenTelemery data to Elastic APM. It's very far from being done or production ready, the code is here (although it's not really documented or so, I mainly just wrote the code to see how far the OpenTelemetry project is).

One thing that comes to my mind is that you could use the OpenTracing API, and since it should be supported by OpenTelemetry (assuming this backwards compatibility really works), the exporter should be able to pick up the data. This would be a good reason to finish the exporter on the branch I linked above and this would be very much aligned with what our latest idea is on how to proceed with OpenTelemetry.

[ryanjshaw]

We can take all the learnings that we had with Java regarding turning OpenTracing format to our Elastic APM format, so that's good.

I just started looking at the APM landscape for the first time today. OpenTracing appears to provide a low barrier to entry for newcomers - understand the API once, then try out different implementations. I want to try out Elastic APM because my client has a support subscription but I'm already struggling to figure out what the equivalent basic concepts are in this world (e.g. "FollowsFrom" reference) - and I imagine everybody who is coming to Elastic APM from OpenTracing is doing exactly the same thing, repeating the effort, and possibly giving up.

Is it not possible to use the https://github.com/opentracing/opentracing-cshar NuGet package, and then write a set of adapters that implements these OpenTracing interfaces (e.g. ITracer, ISpanContext, etc.) with the appropriate apm-agent-dotnet methods? That's how the Jaegar .NET implementation works, I believe.

[gregkalapos]

Hi @ryanjshaw,

I understand that the OpenTracing .NET repo looks nice for newcomers, but please also keep in mind that that project is discontinued and never made it to version 1.

Is it not possible to use the https://github.com/opentracing/opentracing-cshar NuGet package, and then write a set of adapters that implements these OpenTracing interfaces (e.g. ITracer, ISpanContext, etc.) with the appropriate apm-agent-dotnet methods?

Yes, it's possible. We are happy to take contributions to support it, but please also understand it from our point: if we'd do so then we'd put effort to support a library that is version 0.x and will never reach 1.x - that is already publicly announced. So, yes, technically it's possible, but not sure how well time'd be spent on this. Nevertheless, of course if someone from the community wants to build it, I'll be happy to help out.

I'm already struggling to figure out what the equivalent basic concepts are in this world

That's sad to hear - if something is not clear we are more than happy to take feedback and change the documentation. Have you looked at our APM Overview docs? Also the .NET Agent docs is something worth going through.

We also have multiple sample apps here in the repo - this one is one covers lots of things.

Also there are some older posts talking about .NET - those are the announcements, basically all that is still true - 1, 2 and there is also a walkthrough video (not much changed compared to that either - that definitely gives you an overview, and then looking at the official docs you'll always see the most up-to-date things).

Maybe those links help - keep in mind, the docs are always up-to-date, older blogposts may contains APIs that changed, so if something does not work, always look into the docs.

I imagine everybody who is coming to Elastic APM from OpenTracing is doing exactly the same thing.

That's a fair point - this is the first time I hear this is an issue. If this is something we see a lot, we could definitely document this path.

I hope this helps a bit.

[ryanjshaw]

I understand that the OpenTracing .NET repo looks nice for newcomers, but please also keep in mind that that project is discontinued and never made it to version 1. if we'd do so then we'd put effort to support a library that is version 0.x and will never reach 1.x - that is already publicly announced.

Thanks, I didn't realise that - and of course you are absolutely correct! I think what got me is that the OpenTracing bridge page gave me the impression that OpenTracing was something desirable, since it was supported by all the other agents. I understand now that that's just a result of the development history, not a strategy.

That's sad to hear - if something is not clear we are more than happy to take feedback and change the documentation. Have you looked at our APM Overview docs? Also the .NET Agent docs is something worth going through.

Having spent a few more hours on this:

• I didn't realise I was looking at the APM Agent documentation, so I missed the topics in the main documentation like 'Distributed tracing' which would have clued me into the conceptual differences between Elastic APM and OpenTracing and I would have quickly realized that DistributedTracingData allows me to achieve what FollowsFrom did in the OpenTracing API. Then again, now I'm wondering if I'm mistaken, because OpenTracing API implies that FollowsFrom references aren't supported yet DistributedTracingData seems to do the same thing conceptually...

• I note that while the Public API documentation does cover DistributedTracingData, the sections on starting spans and transactions do not mention this optional parameter -- given my earlier mistake of reading the wrong documentation to begin with, I only discovered this data structure when I skimmed through the API a few more times and finally decided to read the scary sounding section on "Manually propagating distributed tracing data"

• I prefer the 'fluent interface' of OpenTracing over the CaptureXXX Elastic APM methods because it made the API more discoverable and plays very nicely with C#8 using declarations - but I quickly built my own e.g.:

using var scope = _tracer
    .BuildTransaction(nameof(GetQueueMessagesAsync), "kafka")
    .WithLabel("topic", _topicName)
    .WithCustom("maxCount", maxCount) // this is an int
    .Start();

var crossGrainMessage = new crossGrainMessage(scope.Transaction.OutgoingDistributedTracingData, …)

I hope this helps a bit.

It did, thanks. Overall the experience of installing Elasticsearch, Kibana, APM, and integrating with the agent has actually been very positive. It looks like I lose 'baggage' from OpenTracing, but I do get metrics and first-class errors...

[megakid]

FYI if you've instrumented in OpenTracing you can get your traces into Elastic via this new server-side Elastic APM feature: https://www.elastic.co/guide/en/apm/server/7.6/jaeger.html

[gregkalapos]

This could be relevant: https://github.com/dotnet/runtime/issues/31373

[bertvan]

@gregkalapos is there any progress/roadmap for the OpenTelemetry APM exporter you POC'ed?

[glucaci]

@SergeyKleyman, @gregkalapos any updates on this?

[gregkalapos]

Not in this repo with the current Elastic .NET APM agent, but the Elastic APM Server has OpenTelemetry support now. Docs here.

[glucaci]

So than this should be closed because will not be implemented. Keeping it open only bring confusion. Also #1234 should be closed with mention that will not be supported.

[AlexanderWert]

https://github.com/elastic/apm-agent-dotnet/issues/1521 implements an OpenTelemetry bridge