Support Distributed Tracing outside of Application Insights

[robece]

Hi team, I have been looking the best approach to implement distributed tracing in durable functions.

I was trying to implement OpenTelemetry using Azure Functions to get all the information in Jaeger or Zipkin like any other .Net Core app using OT, but I had difficulties it seems like it's not supported by internal restrictions (hosted services), I already submitted another issue on this repo OpenTelemetry DotNet repo: https://github.com/open-telemetry/opentelemetry-dotnet/issues/1602.

In addition, I haven't found any updated information about how to implement distributed tracing with durable functions using Application Insights, I found this README but it's using another library Microsoft.Azure.WebJobs.Extensions.DurableTask.Telemetry.

My question to you would be: is this library ready to be use in some specific production environment? or should I wait any integration to start using it in prod?

I just want to confirm if there is any support to work with other cloud native exporters like zipkin or jaeger, I was planning to use OpenTelemetry as I mentioned but just want to discard that alternatives.

I would appreciate any response.

Thanks, RC

[ConnorMcMahon]

Regarding the README you found: the Microsoft.Azure.WebJobs.Extensions.DurableTask.Telemetry was an alpha package for a private preview test of distributed tracing in Durable Functions with Application Insights. We are actually in the progress of merging this into our main dev branch so it will be included in a part of release v2.4.0 of the normal extension Microsoft.Azure.WebJobs.Extensions.DurableTask. Expect that to come out by the end of the month, and you can track the state of that progress here.

At this time, we don't have any explicit plans to support other cloud native exporters at this time, as most of our customers use Application Insights, but the design for our distributed tracing was meant to be flexible, and we would be happy to take community contributions to allow support for other logging providers.

[robece]

Thank you @ConnorMcMahon, then I will wait the next release to replace the alpha library I'm using with the DurableTask.Telemetry extension feature.

Regards, RC

[ConnorMcMahon]

@robece, to clarify: you will no longer need a separate DurableTask.Telemetry package. It will be supported in the mainline Microsoft.Azure.WebJobs.Extensions.DurableTask v2.4.0 release, and you will just need to add a field to your host.json to turn on the feature.

Expect more details in the release notes for v2.4.0.

[davidmrdavid]

Hi @robece ! As of recent, I've started leading the effort of enabling Distributed Tracing for Durable Functions. We're currently tracking the feature, as a whole, in this ticket: https://github.com/Azure/azure-functions-durable-extension/issues/939 . Other smaller tickets may arise from time to time as we address specific limitations and bugs of the current implementation.

As Connor mentioned, the upcoming release of Durable Functions should include an in-progress version of our distributed tracing implementation for Application Insights. I am personally really interested in making this feature flexible enough to let customers use their telemetry tooling of choice, such as the ones you listed.

In the meantime, I would suggest subscribing to the feature-tracking ticket (https://github.com/Azure/azure-functions-durable-extension/issues/939). If possible, I would also appreciate it if you could describe your use-case as a comment in that ticket as well. If other customers end up engaging with it, I would have more leverage to prioritize it :)

That's all for now! Please reach out anytime ⚡ ⚡

[evilpilaf]

At this time, we don't have any explicit plans to support other cloud native exporters at this time, as most of our customers use Application Insights

Isn't this kind of a circular issue, as there's no other choice but to use Application Insights then of course most customers will be using it?

[davidmrdavid]

@evilpilaf : sorry for the late reply here! Yes, I think you're right. I would personally love to design this feature in a way that's amenable with other cloud native exporters. At the moment though, since we're still in the early stages of building this feature, our focus will be on Application Insights. While doing so, I'll keep an eye out for keeping the architecture flexible enough such that, once the feature is more stable, we can consider expanding our dist. tracing clients :)

[davidmrdavid]

More succinctly put: so long as I'm still the main developer leading this effort, you can count that I'll keep an eye out for opportunities to support distributed tracing clients beyond just Application Insights. It's just that, at the moment, our primary concern is to get the feature out of "beta" and into general availability 😃 . The fastest path towards that, for us, is Application Insights.

That being said, these are just my current aspirations. Things could change in the future, just to be clear.

[jvmlet]

Hello @davidmrdavid, any progress with custom exporters ? What is the current status ? And sorry, but I HAVE to ask : OTLP, being THE standard for observability, blessed by Microsoft, why the initial implementation covered App Insights and not OTEL ? There is Azure.Monitor.OpenTelemetry.Exporter, and by implementing OTLP in the first place , you covered ALL current/future/custom providers/vendors/whatever exporters. Why ?

[jvmlet]

Is this pre-requisite ? Can this please be prioritized ? Thanks

[lilyjma]

Hi @jvmlet - thanks for checking in. We do have several customers expressing interest for this feature, and that was why we created the issue you linked. Unfortunately, the scope of the work had to be reduced because of a higher priority effort that the team had. We've edited the issue to reflect this reduction in scope. Once we have more bandwidth, we'll try our best to get to the remaining work to support OT. When that happens, I'll make sure to update this thread. Thanks again for your interest.

[iliya1984]

Hello @davidmrdavid, this feature is very important for my team. Durable function components can not be successfully integrated into the system without a distributed tracing exported to Elastic search via a custom exporter. Is there any progress with this issue ? What is the current status ? Is there planned release date ? It will be great for us if a priority for this issue will be increased !

[davidmrdavid]

@bachuv can you please comment on this ^

[gha-zund]

also for us, this is a big wish! (more a must-have)