search

DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
553 stars 415 forks source link

botocore SQS custom `MessageAttributes` are not being attached to spans #10120

Open miketheman opened 3 months ago

miketheman commented 3 months ago

Summary of problem

Using ddtrace and botocore, we do not see custom MessageAttributes being added to the ddtrace spans.

We added setting custom message attributes via celery kwargs, and can confirm the custom attributed are being emitted to botocore, but are not being expanded or attached to the spans.

Which version of dd-trace-py are you using?

ddtrace==2.10.3

https://github.com/pypi/warehouse/blob/754886cb4e49226dfe75171d2cfec8aff0052ff4/requirements/deploy.txt#L25

Which version of pip are you using?

24.2

Which libraries and their versions are you using?

See https://github.com/pypi/warehouse/blob/754886cb4e49226dfe75171d2cfec8aff0052ff4/requirements/main.txt for full list

How can we reproduce your problem?

Set custom MessageAttributes on a call while being traced https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs/client/send_message.html

For example, "MessageAttributes": {"task_name": {"StringValue": "example_task_name", "DataType": "String"}

What is the result that you get?

Custom MessageAttributes are emitted - but only the _datadog ones get handled.

Here's an example of a logged debug call from botocore in a local development environment:

{"logger": "botocore.endpoint", "level": "DEBUG", "event": "Making request for OperationModel(name=SendMessage) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AmazonSQS.SendMessage', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'Boto3/1.34.134 md/Botocore#1.34.134 ua/2.0 os/linux#6.9.8-orbstack-00170-g7b4100b7ced4 md/arch#aarch64 lang/python#3.12.4 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.134'}, 'body': b'{"QueueUrl": "http://sqs.us-east-1.localstack:4566/000000000000/warehouse-dev-default", "MessageAttributes": {"task_name": {"StringValue": "warehouse.search.tasks.reindex_project", "DataType": "String"}, "_datadog": {"DataType": "String", "StringValue": "{\\"x-datadog-trace-id\\": \\"185612277372681671\\", \\"x-datadog-parent-id\\": \\"4452118822582909169\\", \\"x-datadog-sampling-priority\\": \\"1\\", \\"x-datadog-tags\\": \\"_dd.p.dm=-0,_dd.p.tid=66b3976a00000000\\", \\"traceparent\\": \\"00-66b3976a0000000002936d6202bffdc7-3dc91a7b15e278f1-01\\", \\"tracestate\\": \\"dd=p:3dc91a7b15e278f1;s:1;t.dm:-0;t.tid:66b3976a00000000\\"}"}}, "MessageBody": "eyJib2R5IjogIlcxc2lNbU5oY0hSamFHRWlYU3dnZTMwc0lIc2lZMkZzYkdKaFkydHpJam9nYm5Wc2JDd2dJbVZ5Y21KaFkydHpJam9nYm5Wc2JDd2dJbU5vWVdsdUlqb2diblZzYkN3Z0ltTm9iM0prSWpvZ2JuVnNiSDFkIiwgImNvbnRlbnQtZW5jb2RpbmciOiAidXRmLTgiLCAiY29udGVudC10eXBlIjogImFwcGxpY2F0aW9uL2pzb24iLCAiaGVhZGVycyI6IHsibGFuZyI6ICJweSIsICJ0YXNrIjogIndhcmVob3VzZS5zZWFyY2gudGFza3MucmVpbmRleF9wcm9qZWN0IiwgImlkIjogIjgzYzMxNmE4LWViMjQtNDEyNC1hYzkwLWQzNDE0ZDAzMDViNyIsICJzaGFkb3ciOiBudWxsLCAiZXRhIjogbnVsbCwgImV4cGlyZXMiOiBudWxsLCAiZ3JvdXAiOiBudWxsLCAiZ3JvdXBfaW5kZXgiOiBudWxsLCAicmV0cmllcyI6IDAsICJ0aW1lbGltaXQiOiBbbnVsbCwgbnVsbF0sICJyb290X2lkIjogIjgzYzMxNmE4LWViMjQtNDEyNC1hYzkwLWQzNDE0ZDAzMDViNyIsICJwYXJlbnRfaWQiOiBudWxsLCAiYXJnc3JlcHIiOiAiKCcyY2FwdGNoYScsKSIsICJrd2FyZ3NyZXByIjogInt9IiwgIm9yaWdpbiI6ICJnZW44QDJiNzU2YzA3ODU1ZiIsICJpZ25vcmVfcmVzdWx0IjogdHJ1ZSwgInJlcGxhY2VkX3Rhc2tfbmVzdGluZyI6IDAsICJzdGFtcGVkX2hlYWRlcnMiOiBudWxsLCAic3RhbXBzIjoge30sICJzZW50cnktdGFzay1lbnF1ZXVlZC10aW1lIjogMTcyMzA0NTczOC4xNTUzOTAzLCAic2VudHJ5LXRyYWNlIjogIjM4N2QwZWY0YjllMTQyNDM4YmU3YWJlMGJhNjBkNzlmLWE4YzI4OTUwMjEwYTBiZWIiLCAiYmFnZ2FnZSI6ICJzZW50cnktdHJhY2VfaWQ9Mzg3ZDBlZjRiOWUxNDI0MzhiZTdhYmUwYmE2MGQ3OWYsc2VudHJ5LWVudmlyb25tZW50PXByb2R1Y3Rpb24sc2VudHJ5LXJlbGVhc2U9bnVsbCIsICJoZWFkZXJzIjogeyJzZW50cnktdHJhY2UiOiAiMzg3ZDBlZjRiOWUxNDI0MzhiZTdhYmUwYmE2MGQ3OWYtYThjMjg5NTAyMTBhMGJlYiIsICJiYWdnYWdlIjogInNlbnRyeS10cmFjZV9pZD0zODdkMGVmNGI5ZTE0MjQzOGJlN2FiZTBiYTYwZDc5ZixzZW50cnktZW52aXJvbm1lbnQ9cHJvZHVjdGlvbixzZW50cnktcmVsZWFzZT1udWxsIiwgInNlbnRyeS10YXNrLWVucXVldWVkLXRpbWUiOiAxNzIzMDQ1NzM4LjE1NTM5MDN9fSwgInByb3BlcnRpZXMiOiB7ImNvcnJlbGF0aW9uX2lkIjogIjgzYzMxNmE4LWViMjQtNDEyNC1hYzkwLWQzNDE0ZDAzMDViNyIsICJyZXBseV90byI6ICIyYjY1ODY5MS1jN2JjLTNjMmEtYjM5NS05ZTY3Y2QxNDVkODgiLCAiZGVsaXZlcnlfbW9kZSI6IDIsICJkZWxpdmVyeV9pbmZvIjogeyJleGNoYW5nZSI6ICIiLCAicm91dGluZ19rZXkiOiAiZGVmYXVsdCJ9LCAicHJpb3JpdHkiOiAwLCAiYm9keV9lbmNvZGluZyI6ICJiYXNlNjQiLCAiZGVsaXZlcnlfdGFnIjogIjQ4NDM1NWE5LTYwMTYtNGY0ZS1hZjc2LTI3ZDBlYjZlOTU4ZiJ9fQ=="}', 'url': 'http://localstack:4566/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0xffff83d31880>, 'has_streaming_input': False, 'auth_type': None}}", "thread": 281473237798944}

We can see that both task_name and _datadog keys are added to the MessageAttributes dictionary - so the mechanics to emit them is correct.

What is the result that you expected?

Spans for aws.sqs.sendmessage to have an extra attribute named task_name with the associated string value and can be viewed/searched/excluded for in the Datadog Trace UI.

miketheman commented 3 months ago

I also noticed that the majority of the tests in https://github.com/DataDog/dd-trace-py/blob/e9d838bd358d29b59b9906eb9bb564109743c7b8/tests/contrib/botocore/test.py assert whether the _datadog key either exists or doesn't, but doesn't evaluate whether any of the other custom MessageAttributes make it through?

astuyve commented 3 months ago

Hi @miketheman thanks for reaching out.

Spans for aws.sqs.sendmessage to have an extra attribute named task_name with the associated string value and can be viewed/searched/excluded for in the Datadog Trace UI.

This is an interesting feature request! So we'd take all the k/v pairs from message_attributes and make them span tags?

We can probably do that, I'll see what the team says. In the meantime you can grab the active span and set them yourself if you'd like.

Thanks!

miketheman commented 3 months ago

So we'd take all the k/v pairs from message_attributes and make them span tags?

Exactly!

In the meantime you can grab the active span and set them yourself if you'd like.

That could be interesting, however since I'm using Celery's apply_async on the producer side, I'm not certain I have access to set metadata on spans.