Manual vs Automatic instrumentation

[chazt3n]

Will this method provide less functionality than using the profiler? this seems too good to be true.

Does this mean that by adding Amazon.XRay.Recorder.AutoInstrumentation.Initialize.AddXRay(); I am manually instrumenting my application, or does that mean that I also need to go through the application adding segments/subsegments, and using the AWSXRayRecorder* nuget packages.

Manual Instrumentation

Asp.Net Core

Instead of using profiler, you may choose to manually instrument AWS X-Ray SDK into your Asp.Net Core application. Import AWSXRayRecorder.AutoInstrumentation package into your project. Add the following method into any method in startup.cs or program.cs file Amazon.XRay.Recorder.AutoInstrumentation.Initialize.AddXRay();

[chazt3n]

@lupengamzn - Sorry for the annoying issues. I'm not getting a clear picture of what's required in order to use this. If I can simply add that one line of code (I have cloned/referenced the project) and not use the clrprofiler.dll then that is amazing. But if by doing so, there are more steps then I just need to know which way to go.

[lupengamzn]

Hey @chazt3n ,

Sure, no problem and all issues are welcome!

Will this method provide less functionality than using the profiler?

It provides exactly the same functionality as using profiler.

Does this mean that by adding Amazon.XRay.Recorder.AutoInstrumentation.Initialize.AddXRay(); I am manually instrumenting my application

Yes, this is the only thing you need to do to your application if you choose not to use profiler.

[chazt3n]

@lupengamzn manual (auto) instrumentation is working super well, the only caveat is that background services registered with

AddHostedService<TBackgroundService> are not being traced.

We use background services for guaranteed delivery/exactly once processing of SQS/SNS messages (try not to judge!) and are looking at XRay to trace events across the system.

The background service is guaranteed to be added after the XRay auto instrumentation, but no traces are being generated. Is there a special step for background / hosted services within the web api?

My code basically boils down to this:

  protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {

            while (!stoppingToken.IsCancellationRequested )
            {
                using (var scope = _scopeFactory.CreateScope())
                {
                    var messagesToPublish= await GetMessagesToPublish();
                    foreach (var message in messagesToPublish)
                    {
                            var response = await _snsClient.PublishAsync(request, cancellationToken);                                
                    }
                }
            }
        }

[lupengamzn]

@chazt3n Seems like the background services are executed outside Asp.Net Core's request-response life-cycle, thus will not be captured in X-Ray Agent's use case. In this case, you will need to find out the entry/end point of the background services and call AWSXRayRecorder.Instance.BeginSegment()/AWSXRayRecorder.Instance.EndSegment() (These two APIs are already included in auto instrumentation's library) to manually begin/end segment and the business logic of the background services will be showing up in X-Ray console as a separate graph from the main service.

[lupengamzn]

Based on your code, you may try:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {

            AWSXRayRecorder.Instance.BeginSegment("Background Services");

            // Your business logic within background service
            while (!stoppingToken.IsCancellationRequested )
            {
                using (var scope = _scopeFactory.CreateScope())
                {
                    var messagesToPublish= await GetMessagesToPublish();
                    foreach (var message in messagesToPublish)
                    {
                            var response = await _snsClient.PublishAsync(request, cancellationToken);                                
                    }
                }
            }

            AWSXRayRecorder.Instance.EndSegment();

        }

[chazt3n]

@lupengamzn you are my hero, I really appreciate the guidance - went from total noob to total noob with distributed tracing very fast.

[chazt3n]

@lupengamzn - if AWSXRayRecorder.Instance.EndSegment(); is never called, will tracing data ever be sent to the daemon?

Wondering if I need to be creating/ending these in a loop?

(it's not clear from the Segment Docs)

[lupengamzn]

Hey @chazt3n ,

Thanks for the feedback!

Basically all of the http, AWS or Sql request called within the background services will generate subsegments and there should already be one parent segment to "wrap" them up otherwise it will throw Entity doesn't exist in AsyncLocal exception.

Also, segment has to be ended before it can be further sent to daemon, and the above snippet is just an example of how to trace the background services. You may adjust the location of AWSXRayRecorder.Instance.BeginSegment()/AWSXRayRecorder.Instance.EndSegment() to better fit the business logic of your application.

[chazt3n]

Ok so the last thing I'm trying to figure out is how best to get the traceId out of the SQS consumer (background service)

I can create a parent segment for the entire process which should never end, then within that I can create subsegments, but I need them to have the incoming message's TraceId. So I'm wondering if I should call BeginSegment(name, traceId, parentId) (for theoretically two segments?)

I'm worried about what will happen if I call endsegment there

tricky!

[lupengamzn]

@chazt3n , you may try getting the trace header from AWSTraceHeader as a message system attribute from the SQS message. Here is an example(Java) of how you can get it. Once you have it, you can further extract trace id and parent id from it and then pass to BeginSegment(name, traceId, parentId) to create the parent segment in your background service.

[chazt3n]

OK great that's exactly what I'm doing -

message publisher:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            // Your business logic within background service
            while (!stoppingToken.IsCancellationRequested )
            {
                using (var scope = _scopeFactory.CreateScope())
                {
                    var messagesToPublish= await GetMessagesToPublish();
                    AWSXRayRecorder.Instance.BeginSegment("Background Services");
                    foreach (var message in messagesToPublish)
                    {
                            var response = await _snsClient.PublishAsync(request, cancellationToken);                                
                    }
                    AWSXRayRecorder.Instance.EndSegment();
                }
            }
        }

message consumer:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            // Your business logic within background service
            while (!stoppingToken.IsCancellationRequested )
            {
                using (var scope = _scopeFactory.CreateScope())
                {
                    var messages = await PollSqsForMessages(); //not tracing this due to volume of traces it would generate

                    foreach (var message in messagesToPublish)
                    {
                            AWSXRayRecorder.Instance.BeginSegment("event-consumer", message.GetTraceId()); //pseudocode
                            var response = await CustomCode(request, cancellationToken);
                            AWSXRayRecorder.Instance.EndSegment();
                    }
                }
            }
        }

one question I have is: Are segments required to be unique and should I append a Guid or something? Thank you again for all your support!

[lupengamzn]

@chazt3n , sure, no problem! Segment created by AWS X-Ray SDK library should all be unique and you don't need to do anything about it.