[BUG] 🔭 Better OpenTelemetry traces for incoming backplane notifications

[jodydonetti]

Describe the bug

Community user @imperugo noticed that when using the backplane with OpenTelemetry traces enabled, all the spans for the notifications incoming via the backplane were put under one single parent span, basically creating a single mega-span "containing" all the others.

Ideally, each span for each notification should be on their own.

To Reproduce

Here's a MRE (Minimal Reproducible Example) of the issue:

• setup a backplane
• enable OpenTelemetry
• send some backplane notifications

Expected behavior

The traces for the backplane notifications incoming should be root and not "under" another one.

Versions

I've encountered this issue on:

• FusionCache v1.1.0

Screenshots

(Courtesy of user @imperugo , taken in Grafana Tempo)

[jodydonetti]

UPDATE

While fixing this I noticed another couple of things that, if added to the traces, could make the developer experience better:

• include a tag with the source id (the InstanceId of the remote FusionCache instance)
• change the status of the trace in case of errors, like invalid notifications or similar
• add an event in case of, well, some event occurring during the activity

So yeah, I took this opportunity to make the overall experience better.

[jodydonetti]

Hi all, FusionCache v1.2.0 is out 🥳