Sampling using TraceIdRatioBasedSampler and context propagation - best practices?

[unwork-ag]

Question

Describe your environment.

We are building a distributed system based on Asp.NET Core services and messaging using RabbitMQ. I have configured OpenTelemetry instrumentation for AspNetCore, HttpClient and two additional sources (one for activities in the service and one for message send/receive activities). I configured the built-in TraceIdRatioBasedSampler with 0.0 as the ratio (for testing). In addition I created a custom ActivityListener which is essentially listening to the same sources as OpenTelemetry, uses the same sampling and returns ActivitySamplingResult.None for TraceIds that are filtered out.

What are you trying to achieve?

As far as I understand it is required that the TraceId is routed through all involved systems to ensure that sampling can always make the same decision and no loose spans are reported. This requires context propagation which to some extent relies on the Activity.Current property being set (e.g. when I call into a library that creates activities). However, I observed that ActivitySource.StartActivity returns null in an unpredictable way. Let me explain:

• I receive a RabbitMQ message which has no context information. The extraction through TextMapPropagator.Extract returns a default context. When I pass this into ActivitySource.StartActivity as parent context a non-null Activity object is returned:

                       var parentContext = ea.BasicProperties.ExtractParentContext(); // default context in this case
                       var activityName = $"{ea.Exchange}:{ea.RoutingKey} receive";
                       using (var activity = ActivitySource.StartActivity(activityName, ActivityKind.Consumer,
                                 parentContext.ActivityContext)) 
                        {
                            // activity is not null in this case
                            if (activity != null)
                            {
                                message.InjectParentContext(activity);
                            }

...
                        }

• As shown above this activity is then re-injected into the message (so the receive activity becomes new parent for all further activities) and then the message is put into a channel (System.Threading.Channel) and handled on another thread.
• In the handler (on a separate thread) I then extract the context from the message again and use it as parent context for a new activity:

                       var parentContext = message.ExtractParentContext();

                        using (var activity = ActivitySource.StartActivity(
                                   "myActivity", ActivityKind.Internal,
                                   parentContext.ActivityContext)) 
                        {
                             // activity is null here - propagation of context is broken
                        }

• Now that the activity is null the TraceId is no longer propagated into any nested activities which will create loose spans (not for ratio 0.0 but I have observed it very consistenly for ratio 0.5).
• I could propagate the parentContext object all the way through my call stack and inject it into all StartActivity calls but once I want to call into a library (invoke a HttpClient or whatever) this will not work (i.e. context will not be propagated).
• I have currently created a workaround by having my own ActivityListener return ActivitySamplingResult.PropagationData in case of filtered trace ids. This ensures that activities are always created - but in the end this doesn't look right to me.

What is the proper way to do this?

Additional Context

I know that this could also be related to (mis-?)behavior in the System.Diagnostics library (or any of the built-in instrumentations) but I considered this a more appropriate place.

[xontab]

I am encountering a similar issue that when the sampler is dropping the trace and the ActivitySource.StartActivity returns null. Thus the trace id and other context properties are lost. Is there a better way to still enable the context propagation but limit the traces that are exported?

[mahaisong]

I have a similar problem

[github-actions[bot]]

This issue was marked stale due to lack of activity and will be closed in 7 days. Commenting will instruct the bot to automatically remove the label. This bot runs once per day.

[cfbao]

I was having the same problem, and noticing some popular packages in https://github.com/open-telemetry/opentelemetry-dotnet-contrib (e.g. AWS's) not handling this aspect correctly and ending up with disconnected spans.

However, I saw this note on context propagation today: https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Api/README.md#context-propagation where it asks people to follow the messaging example for context propagation.

In this "messaging example", there's this section of code with comment: https://github.com/open-telemetry/opentelemetry-dotnet/blob/c8a8913152eb4a983df485f583a22ed706c0cbf4/examples/MicroserviceExample/Utils/Messaging/MessageSender.cs#L46-L59

Depending on Sampling (and whether a listener is registered or not), the activity above may not be created. If it is created, then propagate its context. If it is not created, the propagate the Current context, if any.

From what I can tell, this should always work. ActivitySource.StartActivity always creates a non-null activity if it's sampled or if there's no existing Current activity.

---

This is the closest to an official recommendation I can find. I do think this should be highlighted in a more prominent place in the documentation though.