Support or be clear about lack of support for Lambda -> SNS -> SQS -> Lambda

[mikeparker]

I've spent many hours trying to get Lambda -> SNS -> SQS -> Lambda to work in xray, and all my traces end with SNS, never going through to SQS and the next lambda.

My code is not very complicated.

    sess := session.Must(session.NewSession())
    client := sns.New(sess)
    xray.AWS(client.Client)

// do some work

    _, err = client.PublishWithContext(ctx, msg)

AWS documentation says that SNS and SQS should automatically pass on the tracing header to the next lambda, and I can't see anything on my SQS queues configuration to enable tracing, it should just work, right?

I eventually found this comment on the NodeJS library: https://github.com/aws/aws-xray-sdk-node/issues/208#issuecomment-549987195

This is a known issue with propagating trace context through an SQS queue. It is not currently possible to connect a trace passed into an SQS queue to the Lambda consuming it on the other end. This is because although the trace header is propagated in the SQS message, you cannot set it in the Lambda because segments in Lambda are immutable. Enabling this functionality is a feature that is in progress and one we hope to release for all our SDKs. Stay tuned and I will update this issue as we make progress in the Node SDK.

Can you confirm if the golang library has the same issue (due to an underlying issue with Lambda perhaps?) and there is no way of it working (and I am wasting my time trying)?

The only other thing I found was this https://docs.aws.amazon.com/xray/latest/devguide/xray-services-sqs.html which shows some code to manually recreate a segment (Excuse the Java code)

    // Retrieve the trace header from the AWSTraceHeader message system attribute
    String traceHeaderStr = message.getAttributes().get("AWSTraceHeader");
    if (traceHeaderStr != null) {
        TraceHeader traceHeader = TraceHeader.fromString(traceHeaderStr);

        // Recover the trace context from the trace header
        Segment segment = AWSXRay.getCurrentSegment();
        segment.setTraceId(traceHeader.getRootTraceId());
        segment.setParentId(traceHeader.getParentId());
        segment.setSampled(traceHeader.getSampled().equals(TraceHeader.SampleDecision.SAMPLED));
    }

I assume this isn't possible in Lambda for the reasons above (lambda segments being immutable)?

[srprash]

Hi @mikeparker Yes, right now all the Lambda platforms do not support propagation of trace header from SQS to a lambda function. We are hard at work on it.

Can you confirm if the golang library has the same issue (due to an underlying issue with Lambda perhaps?) and there is no way of it working (and I am wasting my time trying)?

Yes, a lambda function with a golang runtime has the same limitation.

I assume this isn't possible in Lambda for the reasons above (lambda segments being immutable)?

Yes, you are right about this as well. And I agree with you that we should document this limitation.

We will be posting updates once this feature is supported. Please stay tuned.

Thanks

[mikeparker]

Thanks for the quick reply @srprash. Good to know i'm not going crazy!

I know its hard to talk about release dates but we need to make a call about whether we're going to roll our own tracing system over the next few months, is this something that might arrive this year? Are we talking months or years roughly?

[srprash]

Hi @mikeparker Since we cannot openly talk about the release date, I have asked the AWS support to reach out to you with the information. Thanks for your patience.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[xrn]

Hi @srprash

Any update on this topic?

[srprash]

Hi @xrn The X-Ray team is working hard to get this support rolled out. We will update this issue once the feature is available. Thanks for being patient.

[dfens1]

thanks, @mikeparker - you saved me hours of wasted efforts!!

[Psykar]

So poking at this a bit, I believe it would be possible to hack this by creating a custom emitter, and using https://docs.aws.amazon.com/sdk-for-go/api/service/xray/#XRay.PutTraceSegments inside the Emit() method on the custom Emitter type.

Note I haven't actually tested this, but seems a plausible workaround to skip the lambda's xray daemon.

type Emitter struct{}

func (e *Emitter) Emit(x *xray.Segment) {
    fmt.Println("segment: ", x)
        // TODO - https://docs.aws.amazon.com/sdk-for-go/api/service/xray/#XRay.PutTraceSegments
}

func (e *Emitter) RefreshEmitterWithAddress(*net.UDPAddr) {}

func stuff() {
    xray.Configure(xray.Config{
        Emitter: &Emitter{},
    })
    ctx, seg := xray.NewSegmentFromHeader(context.TODO(), "somename", nil, &header.Header{
        TraceID:  "traceid",
        ParentID: "parentid",
    })
    defer seg.Close(nil)

    _, subset := xray.BeginSubsegment(ctx, "subseg")
    defer subset.Close(nil)
}

Besides being an awful hack, is there anything fundamentally wrong with this approach?

[willarmiros]

@Psykar that workaround should work, as long as you have X-Ray permissions on your lambda. Just also make sure to attach the correct trace ID, parent ID, and sample decision from the _X_AMZN_TRACE_ID environment variable in order to parent your hacky segment correctly.

[mariotoffia]

Hi @dfens1 is there any progress on this issue - I'm using the Go SDK v2 and would very much like to have the ability to do this or at least Lambda->SQS->Lambda.

(and if possible do Lambda->SQS->Lambda->IoT Core Device Shadow->Kinesis Firehose->Lambda->S3->Lambda 😊))

Cheers, Mario :)

[willarmiros]

@mariotoffia unfortunately the blocking issue for connecting SQS -> Lambda is a fundamental AWS problem not specific to a language, so it is ultimately the same tracking issue as https://github.com/aws/aws-xray-sdk-node/issues/208. This is a very challenging problem that we are working on tackling but unfortunately cannot provide ETAs for.

[mikeparker]

I'm curious why this is so difficult (been waiting over 1.5 years now). I've manually implemented a tracing system before and it was as simple as passing a few guids around as headers or attributes to connect the events together. SQS already has message attributes, why not put just the xray trace id in there? Is the holdup because the team is trying to add every SQS event to xray? Are you trying to avoid using up a message attribute? Are you trying to finalise traces when messages go to dead letter queues? I'm sure most people would be happy with something simple for now (e.g. traces only work on successful runs) even if it has a few caveats to work.

[s1mrankaur]

Any update about this?

+1 for the required support for this feature

[willarmiros]

Hi folks - AWS X-Ray has now officially launched support for linking traces to view SQS -> Lambda traces end-to-end! See https://github.com/aws/aws-xray-sdk-node/issues/208 for details.

That being said, we are aware that the complete flow mentioned in the topic is still not supported (Lambda -> SNS -> SQS -> Lambda) because SNS -> SQS -> Lambda traces will omit the SQS node. I will leave this open to track that issue, but I don't think it'll take quite as long to close this one out :)

[psimsa]

@willarmiros "SNS -> SQS -> Lambda traces will omit the SQS node" - does this mean that simply the SQS step will be missing, or will the tracing chain break completely and new trace id would be generated at the lambda node?

[atshaw43]

The trace chain will not end at SQS. It will be missing. Everything after SQS will show on the trace.

[willarmiros]

Hi folks, today we are pleased to announce that Amazon SNS supports active tracing with AWS X-Ray! This means that you can trace your workflows from SNS->SQS->Lambda with just a couple of clicks. Read here to learn more & get started: https://aws.amazon.com/about-aws/whats-new/2023/02/amazon-sns-x-ray-active-tracing-visualize-analyze-debug-application-performance/

[loi-avela]

Is Lambda->SNS->SQS->Lambda written in Golang supported with XRAY? I enabled tracing on API Gateway,sns, lambda, and I see API/Lambda in the service map traces but no SNS/SQS.