Otel Adding Too Many SQS MessageAttributes?

[Peter-J-Brown]

What version of OpenTelemetry are you using?

Library: opentelemetry/api: "1.1.0" (issue still occurs with 1.2 and 1.0) Lambda layer: arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-5-0:2 (issue still occurs with ver-1-0-0:1 and ver-1-7-0:1)

What version of Node are you using?

14.0.0

What did you do?

I've got a lambda which places messages on an SQS queue, and which is instrumented with open-telemetry. When the lambda places a message on the queue, it attaches two MessageAttributes to the message, but whenever the lambda calls sendMessage(), via the aws-sdk, the lambda otel layer logs the following:

aws-sdk instrumentation: cannot set context propagation on SQS/SNS message due to maximum amount of MessageAttributes

Note: the traces are not being dropped and the messages are successfully placed on the SQS queue.

What's confusing me here is that I know SQS has a limit of 10 MessageAttributes on a message when raw message delivery is enabled (which it is in this case), but I don't understand how I'm getting anywhere near that limit considering that at the point the lambda calls sendMessage(), the message it passes in only has 2 MessageAttributes. I've also checked the message again once it's been put on the queue, and I can't see any new attributes that have been added. Is the otel lambda layer trying to add 8+ MessageAttributes to the message before it's placed on the SQS queue?

What did you expect to see?

No warnings in logs.

What did you see instead?

aws-sdk instrumentation: cannot set context propagation on SQS/SNS message due to maximum amount of MessageAttributes

Upgrading/downgrading both the lambda layer and the otel library don't seem to stop the issues.

Any help would be appreciated, thanks.

[dyladan]

@willarmiros can you take a look at this?

[willarmiros]

Hi @Peter-J-Brown, per the docs on the AWS SDK instrumentation:

Messaging Attributes are added by this instrumentation according to the spec.

According to the messaging spec, it seems like there are 4 producer-specific attributes, and several other generic messaging attributes. However these should only be recorded on the span, not as SQS Message Attributes. @blumamir implemented this MessageAttribute injection, so I'll let him chime in on how many attributes are actually intended to be added. However I think in general we should only be adding 1 for the trace context.

Lastly, I think it's worth adding a configuration to opt-out from adding any message attributes at all, since customers may want to reserve all 10 for their own uses. If @Peter-J-Brown agrees with this approach, we can use this issue to track that as a feature request.

[Peter-J-Brown]

Hi @willarmiros, thank you for the detailed response. I do like the idea of being able to opt-out of adding message attributes altogether, and having the option available would certainly help me with debugging the source of this issue on my end. Thanks again for your help.

[alexandre-bordiere]

I am facing the same problem as @Peter-J-Brown. I have a lambda sending SNS notifications with 1 to 2 attributes maximum.

It seems like OTEL is stacking up attributes if it's being sent by the same lambda instance.

[Peter-J-Brown]

Any update on this?

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[dizer]

In my case library adding 9 attributes, leaving consumer with just one:

[
        "x-amzn-trace-id",
        "traceparent",
        "tracestate",
        "b3",
        "x-b3-traceid",
        "x-b3-spanid",
        "x-b3-flags",
        "x-b3-sampled",
        "x-b3-parentspanid"
]

To reduce amount of attributes you need to update OTEL_PROPAGATORS env variable to something like tracecontext,baggage,xray