FaaS: provide span structure recommendations for multiple layers of instrumentation

[lmolkova]

Azure Functions support in-host and isolated modes, where user code runs in the same (as runtime) or a different process. When users choose isolated mode, host reacts to triggers and then invokes worker process where user code runs. Host knows much more about trigger and can instrument this call properly.

FaaS conventions should suggest if instrumentations should expose some internal details on how they work:

• should both host and worker spans be created? Which kind should worker spans have?
• should call from host to worker be instrumented?
• if worker call is not instrumented and should be hidden from user altogether, how to do it mechanically in OpenTelemtery
• which attributes should be set on which span?

Here're possible approaches:

Option 1: transparent host-worker

• Trigger span (on host) - server/consumer kind
  • Call to worker (client span)
    • Incoming on worker (server span)

Option 2: call to worker process is not traced as if things happen in the same process

• Trigger span (on host) - server/consumer span
  • Incoming on worker (internal span)

Option 3: opaque communication (not possible today):

• Trigger span (on host) - server/consumer span
  • call to worker propagates context and makes it ambient without starting new span

Related: https://github.com/Azure/azure-functions-dotnet-worker/pull/1307

Afaik AWS lambda creates multiple spans today and it'd be great to define a common approach everyone can follow.

[lmolkova]

/cc @trask @RohitRanjanMS

[Oberon00]

I think a main problem here could be that the "Incoming on worker" span might extract a parent in an application-specific way, then, similar to messaging scenarios, you could have the application-level "sender" or "client" that you might want to use as parent and the technical caller (e.g. host span).

Additionally, I think some cloud providers already create spans for internals but only expose them with their own tracing solutions.

[cartersocha]

@lmolkova could you or Rohit join the sig to discuss this one day? We'd love to learn more about the scenarios yall are targeting

[RohitRanjanMS]

@lmolkova could you or Rohit join the sig to discuss this one day? We'd love to learn more about the scenarios yall are targeting

I am OOF but I can join the 2/28 SIG meeting.