SNS to SQS distributed tracing

[dcfranca]

I see that this feature is supported on Python and NodeJS, is it supported in Go?

There is a similar issue closed on the dd-trace-js: https://github.com/DataDog/dd-trace-js/issues/1280 And the documentation mentions Python and NodeJS: https://www.datadoghq.com/blog/aws-serverless-tracing-datadog-apm/

I have an application that I'm trying to implement a distributed tracing for the same scenario:

I'm wrapping the SNS client in DD AWS Trace

    sess = awstrace.WrapSession(sess)

    ing.snsClient = sns.New(sess)

Then, passing the context:

    if _, err := ing.snsClient.PublishWithContext(ctx, &sns.PublishInput{
        MessageAttributes: attr,
        Message:           aws.String(string(bytes)),
        TopicArn:          aws.String(ing.SNSTopicArn),
    }); err != nil {
        ing.logger.Error("failed to send event,", err)
        return http.StatusInternalServerError, pkgerrors.Wrap(err, "failed to publish message")
    }

And then wrapping the SQS client, and receiving the context

    sess = awstrace.WrapSession(sess)
    sqsClient := sqs.New(sess)

            output, err := sqsClient.ReceiveMessageWithContext(ctx, &sqs.ReceiveMessageInput{
                QueueUrl:        aws.String(d.env.QueueURL),
                WaitTimeSeconds: aws.Int64(defaultWaitTimeSecs),
            })

However, the trace stops at publishing to SNS, doesn't follow the e2e process as I would like to

So, in the tracing I can see the request flowing from the http request to the SNS publishing, but nothing else.

The body of the message looks like this:

{
  "Type" : "Notification",
  "MessageId" : "XXXXXXXX",
  "TopicArn" : "arn:aws:sns:xxxxx",
  "Message" : "{\"specversion\":\"1.0\",\"id\":\"1\",\"source\":\"mysource\",\"type\":\"myevent\",\"subject\":\"k6\",\"datacontenttype\":\"application/json\",\"data\":{\"mydata\":\"test\"}}",
  "Timestamp" : "2023-03-10T16:41:53.391Z",
  "SignatureVersion" : "1",
  "Signature" : "XXXXXX,
  "SigningCertURL" : "https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-XXX.pem",
  "UnsubscribeURL" : "https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=XXXX",
  "MessageAttributes" : {
    "subject" : {"Type":"String","Value":"abc"},
    "source" : {"Type":"String","Value":"source"},
    "type" : {"Type":"String","Value":"event"}
  }
}

And nothing on the message attributes

Is it possible to implement it in Go? If so, how?

[dcfranca]

Anyone?

[ajgajg1134]

Hello! I've shared this ticket with the Serverless team at Datadog who previously created that support in Python/NodeJS. It sounds like this isn't currently supported today directly, but they'll be noting down this as a feature request. If you'd like to attach your orgs information to that feature request feel free to open a ticket with Datadog Support

[darccio]

@purple4reina @nhulston If I'm not mistaken, PR #2917 should fulfil this request.

[nhulston]

@purple4reina @nhulston If I'm not mistaken, PR #2917 should fulfil this request.

@darccio This PR only injects trace context for SQS/SNS producers. There is no logic in dd-trace-go for SQS/SNS consumers to extract trace context. This is all we need to get distributed tracing to work in Lambda, since trace extraction in Lambda is handled by the Datadog Lambda Extension.

This will not fully work in non-serverless cases, since consumer code isn't implemented yet.

To get distributed tracing support in Lambda, users will need to wrap their function handler and call AppendMiddleware:

import (
        awstrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/aws/aws-sdk-go-v2/aws"
)

func main() {
    lambda.Start(ddlambda.WrapFunction(FunctionHandler, nil))
}

func FunctionHandler(ctx context.Context, input LambdaInput) (int, error) {
    cfg, err := config.LoadDefaultConfig(ctx)
        if err == nil {
                awstrace.AppendMiddleware(&cfg)
        }
}

[darccio]

@nhulston Thanks for the clarification!

[darccio]

@dcfranca Does it the provided fulfil your request?