Distributed Tracing Not Linking to Parent Function in Isolated Worker Mode

nathan-roe-adesa commented 7 months ago

Description

With a function app deployed in process, the distributed tracing telemetry links the start of the orchestration to the function request that started the orchestration like this:

When we deploy the application in isolated worker mode, the orchestration telemetry is no longer correlated to the function request where it started. We still get the durable orchestration and activity telemetry but not the link to the parent request where the orchestration started.

Expected behavior

The orchestration telemetry should be linked (by the Parent ID property) to the function request that started the orchestration.

Actual behavior

The orchestration appears telemetry appears to only be linked to itself:

Known workarounds

None

App Details

Microsoft.Azure.Functions.Worker.Extensions.DurableTask 1.1.1 V4 function runtime C# .NET 8

If deployed to Azure

Timeframe issue observed: After deploying to isolated worker
Function App name: cvn-rti-east-uat
Function name(s): Function OpenLaneRtiFunction starts orchestration named VehicleChangedOrchestrationFunction
Azure region: East US
Orchestration instance ID(s): c3bbc081e7c24485af1c57f2595711ca
Azure storage account name: qfknic67jj3dgazfunction

davidmrdavid commented 7 months ago

FYI @bachuv

GillesTourreau commented 4 months ago

Do we have an idea when this issue will be fixed ? It is very hard to understand in the traces the origins of the orchestration start (timer, http,...) !

davidmrdavid commented 4 months ago

@GillesTourreau not aware of an ETA at the moment, but it seems to be tracked in the Distributed Tracing GA work item list (https://github.com/Azure/azure-functions-durable-extension/issues/2660) so it is on our radar

RobARichardson commented 4 months ago

I noticed this same issue back in November of 2023 and created #2662.

It is very hard to understand in the traces the origins of the orchestration start (timer, http,...) !

Agree with @GillesTourreau. This is still a big pain point from an observability and troubleshooting perspective.

NickSevens commented 1 day ago

Hi, is there any update regarding V2 distributed tracing in Isolated Worker model?

cgillum commented 1 day ago

Not yet, but it's on our backlog and we plan to have this fully working by the end of the current semester (ending Spring of next year).

Azure / azure-functions-durable-extension