Duplicate App Insight Telemetry

Gilligan128 commented 3 years ago

Description

Azure Function runtime and Durable SDK seem to be emitting two different Request and Dependency telemetries for a given Orchestrator +activity execution.

Expected behavior

for a given Orchestrator+Activity+Client execution, seeing only one Request and Dependency combo (one Orch request, one Orch->Activity dependency, one Activity-> SQL dependency) in a single, correlated trace, with cloud role name and operation name set.

Actual behavior

for a given Orchestrator+Activity execution, I see two Request and dependency telemetries per execution. One is correlated between Orchestrator and Activity, e.g they share an operationid. But it has no cloud role name nor operation name. Nor is it customizable via Initializers. The other is not correlated but has clodu role name and operation name. It is also customizable via initializers (although Activity.Current doesn't match in the Initializer)

Relevant source code snippets

host.json:

{
  "version": "2.0",
  "extensions": {
    "durableTask": {
      "tracing": {
        "distributedTracingEnabled": true,
        "traceInputsAndOutputs": true,
        "DistributedTracingProtocol": "W3CTraceContext"
      }
    }
  },
  "logging": {
    "applicationInsights": {
      "samplingExcludedTypes": "Request",
      "samplingSettings": {
        "isEnabled": true
      }
    }
  }
}

Additional source code and Console output logs can be made available upon request but I cannot publicly share it.

Known workarounds

none

App Details

Durable Functions extension version (e.g. v1.8.3): 2.4.3
Azure Functions runtime version (1.0 or 2.0): 2.0
Programming language used: C#

Screenshots

Privately available upon request.

If deployed to Azure

We have access to a lot of telemetry that can help with investigations. Please provide as much of the following information as you can to help us investigate!

Timeframe issue observed:
Function App name:
Function name(s):
Azure region:
Orchestration instance ID(s): MachineDataOrchestrator
Azure storage account name:

cgillum commented 3 years ago

FYI @bachuv and @TsuyoshiUshio. This might be a consequence of the design where DTFx telemetry and Functions telemetry are separate.

Gilligan128 commented 3 years ago

That is what I figured. And that would be fine IMO if:

I could customize the DTFx telemetry. I have looked into every avenue to do this short of hijacking the private TelemetryClient in TelemetryActivator via reflection. Or at least ensure proper cloud role and operation names.
I can turn off the Function telemetry (but only for Orch/Entity/Activity functions) This may be possible and I just don't know how.

And I say overall my teams and I love using Durable functions.

ConnorMcMahon commented 3 years ago

@Gilligan128, I believe you can turn off functions telemetry on a function-by-function basis via the category Function.<function-name>. That is obviously not ideal here, but may be a solid workaround in the meantime?

To clarify on your requirements, you would need the ability to fix the DTFx telemetry with the proper data fields and remove duplicate function telemetry right?

TsuyoshiUshio commented 3 years ago

Yes. This is known issue. I already submit a pull request for WebJobsSDK side for enabling filtering and make the custom property avaiable. https://github.com/Azure/azure-functions-durable-extension/tree/dev/samples/correlation-csharp#limitation We can add it for the Durable Extensions side.

Gilligan128 commented 3 years ago

Oh nice, @TsuyoshiUshio ! Thanks for responses!

ConnorMcMahon commented 3 years ago

Adding @bachuv for reference here.

Azure / azure-functions-durable-extension