SQS Tracing with AWSTraceHeader

[rogueai]

Hi,

I have an SQS queue subscribed to an SNS topic and in turns triggers a Lambda on new SQS messages.

I've tried following the docs here: https://docs.aws.amazon.com/xray/latest/devguide/xray-services-sqs.html to create segments related to SQS, but so far everything I tried didn't work. I've added a segment this way:

const traceHeaderStr = record.attributes.AWSTraceHeader;
const traceData = AWSXRay.utils.processTraceData(traceHeaderStr);
const segment = new AWSXRay.Segment("SQS", traceData.Root, traceData.Parent);
delete segment.service;
segment.origin = "AWS::SQS";
segment.inferred = true;
segment.addPluginData({
   operation: "SendEvent",
   region: record.awsRegion,
   request_id: context.awsRequestId,
   queue_url: record.eventSourceARN
});

What this produces though is something like this:

As you can see, the SQS segment is created with SNS as a parent, however the Lambda invocation is disconnected.

I've also tried other approaches that didn't work:

• overriding the _X_AMZ_TRACE_ID env variable with the AWSTraceHeader, before initializing the XRay SDK
• setting the tracing to "PassThrough" and creating segments manually

Could you advise a possible workaround, to get this working?

Thank you!

[willarmiros]

Hi @rogueai, This is a known issue with propagating trace context through an SQS queue. It is not currently possible to connect a trace passed into an SQS queue to the Lambda consuming it on the other end. This is because although the trace header is propagated in the SQS message, you cannot set it in the Lambda because segments in Lambda are immutable.

Enabling this functionality is a feature that is in progress and one we hope to release for all our SDKs. Stay tuned and I will update this issue as we make progress in the Node SDK.

[dsharkou]

Hi @willarmiros, Do you have any approximate date for release this feature? We are also need this functionality (in my case sqs trigger lambda which send message to ses).

[willarmiros]

Unfortunately we cannot provide dates for upcoming features. However this feedback is appreciated so we and the other teams involved in the change can prioritize the feature. Stay tuned for updates!

[AlexRomashevski]

Hi @willarmiros,

Also, vote to move this feature on the top for your team backlog. Our team also waiting for this improvement to have a clear map on X-Ray. I hope it will not take to much to move it on live.

Thank you!

[TLaue]

+1

[dimmy-timmy]

+1. I have the same issue on my project

[asaghri]

+1

[misterjoshua]

+1. I am also struggling with this limitation.

[RyanV]

+1

[billybonks]

would love to have this aswell, it would signficantly improve the value of xray.

[danilofuchs]

We also really want to see a more connected map. We use this Lambda -> SNS -> SQS -> Lambda across 3 serverless applications, and X-Ray would enable us to monitor the entire flow

[JohanZackrisson]

+1

[joshgoodwin]

+1

[lgalumbres]

+1 Would love to see this feature supported as well as this would provide significant insight on the traces for our message processing workflow.

[alexmnyc]

this is needed greatly! otherwise, you cannot use xray with lambdas effeciently

[awssandra]

Thanks everyone, for the +1! The X-Ray Service Team is hard at work on this. We hope to post an update on this soon - thank you for you continued patience.

Stay healthy and safe! Sandra

[codergeorgej]

Hi,

I have an SQS queue subscribed to an SNS topic and in turns triggers a Lambda on new SQS messages.

I've tried following the docs here: https://docs.aws.amazon.com/xray/latest/devguide/xray-services-sqs.html to create segments related to SQS, but so far everything I tried didn't work. I've added a segment this way:

const traceHeaderStr = record.attributes.AWSTraceHeader;
const traceData = AWSXRay.utils.processTraceData(traceHeaderStr);
const segment = new AWSXRay.Segment("SQS", traceData.Root, traceData.Parent);
delete segment.service;
segment.origin = "AWS::SQS";
segment.inferred = true;
segment.addPluginData({
   operation: "SendEvent",
   region: record.awsRegion,
   request_id: context.awsRequestId,
   queue_url: record.eventSourceARN
});

What this produces though is something like this:

As you can see, the SQS segment is created with SNS as a parent, however the Lambda invocation is disconnected.

I've also tried other approaches that didn't work:

• overriding the _X_AMZ_TRACE_ID env variable with the AWSTraceHeader, before initializing the XRay SDK
• setting the tracing to "PassThrough" and creating segments manually

Could you advise a possible workaround, to get this working?

Thank you!

@rogueai How did you manage to get SNS -> SQS to come up in the servicemap ? My request tracing gets stopped at SNS. (SQS doesnt showup on the service map)

[pdeak]

It feels to me like the "enhancement" label on this issue is debatable. From what I can see, this means that it is impossible to use X-Ray for its primary intended purpose (transaction visualisation across services) when using SQS and Lambdas?

It doesn't look like there's any kind of workaround or hack to get around this in the meantime, so it essentially makes X-Ray unusable if your architecture just happens to include Lambdas and SQS? I would have thought this would constitute a somewhat more serious defect...

[danieletieghi]

+1

[stockf]

+1

[azambrano]

+1

[willarmiros]

Hello @pdeak, This issue is marked as an enhancement because the X-Ray team working to expand X-Ray's functionality in several use cases. Unfortunately it is not as simple as a missing link due to a bug. We intend to address this in the near future.

[dfens1]

+1

[austin-johansen]

+1

[josuelima]

+1

[5t33]

+3

[5t33]

@rogueai I'm actually curious how you even got the SNS -> SQS trace that I see in your graph. I've been unable to get my trace header to propagate through sns to sqs.

[anaida-gasparyan]

@5t33 I also wonder how @rogueaimanaged to have that trace. I am struggling for a while to have SNS -> SQS trace. We are using Java Spring AWS SDK. Is there a way to propagate the traceid header from SNS to SQS?

[rogueai]

@5t33 @anaida-gasparyan Unfortunately, many moons have passed since I last looked into that, I don't have access to the original code anymore.

If I recall correctly, at some point I was trying to manually construct segments and sending them to xray, so that SQS you see in the picture might as well be one of my "hacks". In fact, you can see the suspicious "remote" tag on the node (not the usual aws service tag) so that was probably me filling in the segment/subsegment details.

Good luck!

[kfcobrien]

+1

[AndresSomadossi]

+1 We're still waiting for this feature

[rcrelia]

+1

[aflinton]

+1

[micheleriso-emis]

+1

[orfeu65]

+1

[jstallm]

Enabling this functionality is a feature that is in progress and one we hope to release for all our SDKs. Stay tuned and I will update this issue as we make progress in the Node SDK.

@willarmiros , thank you for the hope that this feature is coming. Its been almost 1 year since this answer. As of today, is it still true that AWS is adding this feature to the aws sdks? If so, please update since its been a while since this issue was created first. Just wanting to make sure this is still on the list of slated improvements, and it would be great to know if this is coming to the dotnet core sdk as well.

It looks like this is still issue today

[Crafoord]

+1 We are also trying to get this working! Serverless architecture really needs this to be reliable!

[ItayGoren]

+1

[rkatsyuryna]

+1

[kelesaoui]

+1, we still waiting for this enhancement, otherwhise, serverless application miss reliability tracing. khalid

[flochaz]

Here is a workaround : https://github.com/flochaz/x-ray-lambda-sqs-lambda where I

1. disable Xray on consumer lambda
2. Rebuild the segments (SQS and lambda one) manually using the SDK: https://github.com/flochaz/x-ray-lambda-sqs-lambda/blob/main/lib/lambda/consumer-function/index.js

PS: the code is setting a 10s delay on the sqs message (producer lambda code) and 2 second sleep in consumer lambda to highlight proper timing

[jstallm]

@flochaz , can you please summarize the workaround in this thread to avoid forcing developers to dig through the code to discover? Please. Thanks for the workaround solution

[joshgoodwin]

Here's the actual AWS repo that has the example code:

https://github.com/awslabs/aws-solutions-constructs/tree/master/source/patterns/%40aws-solutions-constructs/aws-sqs-lambda

Of course, it's typescript 🙄

[johanneswuerbach]

Is the recent release "AWS X-Ray now supports trace context propagation for Amazon Simple Storage Service (S3)" https://aws.amazon.com/about-aws/whats-new/2020/11/aws-x-ray-supports-trace-context-propagation-amazon-simple-storage-service-s3/ somehow affecting this issue

From the release post it sounds like X-Ray would connect requests from lambda ~> s3 ~> s3 event notification ~> SNS ~> SQS ~> lambda, but it still doesn't seem to work despite lambda being mentioned in all the docs 🤔 .

[benkeil]

+17

[tomaszdudek7]

Yet another +1.

[johncade]

+1

[mabecth]

This seems to be a problem still. Is the workaround still viable or is there a better way now?

[brcourt]

Ran into this issue as well, looking forward to this feature being released. In addition, would like to see similar trace spanning through DynamoDB streams.

[mattopsh]

+1