search

Azure / azure-functions-durable-extension

Durable Task Framework extension for Azure Functions
MIT License
711 stars 263 forks source link

V2 Distributed Tracing Showing Long Response Times when Extended Sessions is enabled #2749

Open nathan-roe-adesa opened 4 months ago

nathan-roe-adesa commented 4 months ago

Discussed in https://github.com/Azure/azure-functions-durable-extension/discussions/2747

I'm moving this to an issue because disabling distributed tracing made the issue with slow executions go away. It seems like enabling tracing causes the application to execute differently than with the tracing off.

Originally posted by **nathan-roe-adesa** February 20, 2024 I enabled V2 distributed tracing in one of my applications and I'm seeing long response times being logged that don't seem to make sense because the functions themselves are executing in 10's of milliseconds. See the screenshot below: ![image](https://github.com/Azure/azure-functions-durable-extension/assets/132937001/7badc78e-2091-4975-bd18-05b808fa0423) Any idea why this is happening?
nathan-roe-adesa commented 4 months ago

Interesting graph from the function metrics in Azure Portal

image

This is a graph from 8 AM to 5 PM EST on 2/21/24. I deployed the change to turn on distributed tracing around 11:30 AM and I turned it off around 3:30 PM. You can see a distinct difference in the graph when it is on versus off. With tracing on it seems that a few of the instances are handling most of the requests whereas when it's off the load is balanced across all the instances.

nathan-roe-adesa commented 4 months ago

Turning off extended sessions seems to have resolved the issue for me. It's not clear to me what the link is between extended sessions and distributed tracing, but there was definitely a major regression to my application when they were both enabled at the same time.

lilyjma commented 4 months ago

@bachuv - FYI