Multiple services

[thrixton]

Question

I have multiple services in a various assemblies and I'd like to be able to track each one by name

In the code below, there is the option to call AddService multiple times but I cannot see how to get a mapping from that to a service specific ActivitySource
Startup .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(info.name)

E.g. Lets say I have a UserService and a Repository<User> service
I'd like to add add a services UserService and Repository_User and then in my UserService I'd have an ActivitySource named UserService and in Repository<User> I'd have an ActivitySource named Repository_User

Is this configuration possible?

I'm using Jaeger exporter if it makes a difference

[cijothomas]

Not fully sure if I understood the qn.. You can create more than TracerProvider, each with its own Resource, and each listening to separate ActivitySources. Is that what you were looking for?

[thrixton]

Yes that sounds what I'm after, do you have an example?

On Tue, 11 May 2021, 02:30 Cijo Thomas, @.***> wrote:

Not fully sure if I understood the qn.. You can create more than TracerProvider, each with its own Resource, and each listening to separate ActivitySources. Is that what you were looking for?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/open-telemetry/opentelemetry-dotnet/issues/2040#issuecomment-836926094, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDJMQ32MBSPJYBRWIN6SPTTNACZRANCNFSM44QV6JNQ .

[AlexGS74]

I'd second that request. Just a quick demo how to create the second TracerProvider would be very helpful.

[AlexGS74]

Would it be something like this:

var openTelemetry = Sdk.CreateTracerProviderBuilder()
                    .AddSource("Samples.SampleClient", "Samples.SampleServer")
                    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("otlp-test"))
                    .AddOtlpExporter(opt => opt.Endpoint = new Uri(endpoint))
                    .Build();

Where the AddService defines the span service and AddSource will define what ActivitySource will be traced by this Tracer. Would this be correct approach?

[thrixton]

Would it be something like this:

var openTelemetry = Sdk.CreateTracerProviderBuilder()
                    .AddSource("Samples.SampleClient", "Samples.SampleServer")
                    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("otlp-test"))
                    .AddOtlpExporter(opt => opt.Endpoint = new Uri(endpoint))
                    .Build();

I use this pattern currently, and it has the downside of every operation being grouped / tagged under the service otlp-test, instead of (e.g. Samples.SampleClient, Samples.SampleServer).

If I add multiple services(..AddService("Service1").AddService("Service2"), it appears that the last one wins, e.g. everything is grouped under that name.

I cannot find a way to map a service name back to a source, in my desired state it is a 1=1 mapping.

I think the .NET implementation (at least) is assuming that a process is a service, whereas what I want is the ability to separate my process into multiple logical services.

Using Jaeger at least, logically separating services has the benefit of colour coding each service which makes it easy to visually distinguish logical components of a larger application. Maybe a picture helps :)

[AlexGS74]

I mean that block is a way to create a tracer. I'd call it twice with one tracer setup to listen to the "Samples.SampleClient" source and reporting the spans as "Samples.SampleClient" service, and the other one to cover the second case:

var openTelemetry1 = Sdk.CreateTracerProviderBuilder()
                    .AddSource("Samples.SampleClient")
                    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("Samples.SampleClient"))
                    .AddOtlpExporter(opt => opt.Endpoint = new Uri(endpoint))
                    .Build();

var openTelemetry2 = Sdk.CreateTracerProviderBuilder()
                    .AddSource("Samples.SampleServer")
                    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("Samples.SampleServer"))
                    .AddOtlpExporter(opt => opt.Endpoint = new Uri(endpoint))
                    .Build();

This is what I'm doing now.

[thrixton]

@AlexGS74 I see now, does that work? e.g. correlate the telemetry for the sources, Samples.SampleClient -> Samples.SampleClient and Samples.SampleServer -> Samples.SampleServer

[AlexGS74]

Yes, it works. Each of the tracers will result in a separate service name in the spans it produce.

[buvinghausen]

@AlexGS74 I have multiple services running in a single hosting process but each service has it's own ASP.NET Controllers, HttpClients, MongoDB database, and uses NServiceBus to handle the messaging. I get the notion of spinning up the multiple tracer providers but how does the instrumentation discern which service is the point of origination?

I definitely would rather have my service names be Billing.Web, Messages.Web, Documents.Web, etc than just currently Hosting.Web.

[AlexGS74]

Only case with which I still struggle with is for a background process. Currently what happens is the background process' spans get on to a request's traceId that caused the process to spin up. All other cases, like request/response with context propagation seem to work just fine. To fix my case I need to figure out a way to create a parent span with no context. ATM it's not obvious how to do that.

[buvinghausen]

@AlexGS74 thanks for following up. For request/response I'm assuming you have an action filter that starts an activity from the service specific activity source?

[buvinghausen]

@AlexGS74 you can disregard the prior questions it looks like I just configure the AspNetCoreInstrumentation to filter out requests from other services and that solves that riddle. Now I just need to ping my main man Jimmy Bogard and see if we can get a similar filter setup for the NServiceBus side for my worker process and this problem will be solved for good. Many thanks for pointing me in the right direction!

[warning-explosive]

@AlexGS74 you can disregard the prior questions it looks like I just configure the AspNetCoreInstrumentation to filter out requests from other services and that solves that riddle. Now I just need to ping my main man Jimmy Bogard and see if we can get a similar filter setup for the NServiceBus side for my worker process and this problem will be solved for good. Many thanks for pointing me in the right direction!

@buvinghausen could you show your filtering configuration please? I'm trying do the same thing - configure multiple services with their own controllers in single process. In the end I either receive duplicates in tracing backend or receive nothing. I use this callback from documentation https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry.Instrumentation.AspNetCore#filter . If you return false from this callback it sets false to activity.IsAllDataRequested property and skips all subsequent callbacks.

[github-actions[bot]]

This issue was marked stale due to lack of activity and will be closed in 7 days. Commenting will instruct the bot to automatically remove the label. This bot runs once per day.