How to prevent incoming messages from forming an endless span hierarchy?

[object]

For each incoming message for an actor Phobos creates a child span, so if this is a long living actor, its incoming messages forms an endless trace transaction. This may not be suitable for a worker actor (e.g. routee executing commands). A simple illustration is ConsoleReaderActor from the Akka.NET Bootcamp. Each word typed on a keyboard can be considered as a transaction root that starts with a message to ConsoleReaderActor and ends with a message to ConsoleWriterActor. Instead all word input becomes a part of a single long trace consisting of 2 * N spans where N is a number of words typed.

It would be very useful if there was a way to configure how incoming messages affect trace span hierarchy. Messages of certain types may be treated as siblings while other messages become children of the current trace span.

[Aaronontheweb]

It's very difficult for us to tell, automatically, whether or not operations should be treated as child or siblings - i.e. the Links you can customize on a given span. Probably the best way to handle this is to:

1. Exclude the messages using ITraceFilter, even if they have an active SpanContext inside of them;
2. Grab the Context.GetInstrumentation().UsableContext - this lets you access any SpanContext that was included in a trace sent to the actor, but was not incorporated into a new ActiveSpan. This is what we use for doing things like propagating trace over /system actors that don't have tracing enabled, such as all of the Akka.Remote actors. This value isn't publicly exposed on the IPhobosActorContext but we could easily change that.
3. Manually create a new child span using that UsableContext as either a parent or sibling span.

That would give you full control over what the span hierarchy looks like since Phobos can't make that level of domain-specific determination for you. Does that idea sound feasible in practice?

[object]

Thanks @Aaronontheweb. I was not aware of UsableContext. It might work. I will book an hour with you so we can talk about this in depth.

[Aaronontheweb]

Hi Vagif, please give Phobos 2.0.3-beta1 a try - it has a new setting for disabling the automatic creation of child spans upon receive. You can see the instructions on how to use this feature here https://phobos.petabridge.com/articles/releases/RELEASE_NOTES.html#203-beta1-may-12th-2022 - along with code for being able to manually extract the sender's context and use it to create your own spans going forward.

[object]

@Aaronontheweb I am trying Phobos 2.0.3-beta1. If I disable automatic creation of child spans they are no longer created. But I am having problem of creating my own, probably because I don't have this code in my actors:

Receive<GetActiveSpan>(_ =>
    {
        // should be default(TelemetrySpan) when running with
        // phobos.tracing.create-trace-upon-receive = off
        Sender.Tell(new SpanResponse(Context.GetInstrumentation().ActiveSpan));
    });

I don't find any traces of GetActiveSpan message. Didn't find it in Phobos namespaces, nor in OpenTelemetry ones. Where is it defined?

And if this is a missing link, then I assume this piece of code should be injected in all actors that want to create their own spans, right?

I also didn't understand the comment "should be default(TelemetrySpan) when running with phobos.tracing.create-trace-upon-receive = off", because the sample code doesn't return default(TelemeterySpan). Should it be defalut when create-trace-upon-receive is enabled?

[Aaronontheweb]

GetActiveSpan was just a sample message from my unit test - it's not specific to Phobos itself.

The real piece of code you need to create your own spans is this (from the other sample in the docs):

using (var mySpan = Context.GetInstrumentation().Tracer.StartActiveSpan("MyMsg", SpanKind.Consumer,
                   Context.GetInstrumentation().UsableContext ?? default))
        {
            _logging.Info("My event");
            Sender.Tell(new MyReply("reply"));
        }

Context.GetInstrumentation().UsableContext is what will allow you to access any SpanContext that has been propagated to you. Please let me know if that helps!

[object]

Thanks @Aaronontheweb. Still stuggling to create my own spans, but I need to investigate it more. Will get back to you once I know more.

[object]

In general this new Phobos feature addresses my issue so I am closing this one. I will open a new issue to discuss a couple of details of how this feature works.