Cannot enable DD_MERGE_XRAY_TRACES without also enabling XRay tracing

[JJSphar]

Expected Behavior

To be able to enable serverless trace merging between Step Functions (Xray traced) and Lambdas (ddtrace'd).

Specifically, the ability to set DD_MERGE_XRAY_TRACES=true in the environment variables of a lambda. My template consists of a Step Function that triggers a lambda. I'd like to trace the SFN with Xray, and use ddtrace for my python lambda.

Actual Behavior

The DataDogServerless macro does not provide a way to independently configure the DD_MERGE_XRAY_TRACES value, without enabling both XRay AND ddtrace for a lambda.

If ddtrace is enabled for a lambda, the following line will run https://github.com/DataDog/datadog-cloudformation-macro/blob/21047dffdc4c1bc1094031c5b109f7e8e29f2dc8/serverless/src/tracing.ts#L128

This will cause the "DD_MERGE_XRAY_TRACES" env var to be overridden with the value of tracingMode == Hybrid. Hybrid mode would require enabling duplicate Xray and DDTrace functionality, which I don't want to do.

Steps to Reproduce the Problem

1. Create a CFN template with an AWS::Serverless::Function, an AWS::Serverless::StateMachine, and the DatadogServerless macro
2. Enable XRay tracing for step machine in cloudformation, enable ddtrace for lambda via default macro settings
3. To enable serverless trace logging, set DD_MERGE_XRAY_TRACES=true in the environment variables block of the lambda
4. Deploy the stack using SAM
5. Inspect CloudFormation to see the processed template has "DD_MERGE_XRAY_TRACES=false"

Specifications

For the Datadog Serverless Macro:

• Deployment method and version (CloudFormation, SAM, CDK): SAM
• Datadog CloudFormation Macro version: 0.2.5
• Lambda function runtime (Python 3.7, Node 10, etc.): Py39

Stacktrace

N/A

Is there a way I can forcibly enable XRay trace merging, without enabling XRay for all my functions? Could we surface "mergeXrayTraces" as a first-class parameter for this macro?

[tianchu]

@JJSphar Thanks for creating this issue! I think it might be worth moving this into a FR with the Datadog support team. It isn't a limitation by the macro itself, there isn't official support for merging non-lambda X-Ray traces with DD Lambda traces. DD_MERGE_XRAY_TRACES was originally designed to connect X-Ray Lambda traces (not step functions traces) with DD Lambda traces.

That said, we are working on tracing Step Functions directly and connecting with Lambda traces. Perhaps you could give some thoughts to our product team by filing a FR or having a conversation? 🙇

[JJSphar]

Hi @tianchu, thanks for your quick response. I have one question:

In the docs page I linked (Serverless Trace Merging) the first "Use Case" seems to be exactly what I'm trying to accomplish:

In an AWS serverless environment, you are already tracing your Lambda functions with dd-trace, you require AWS X-Ray active tracing for AWS managed services such as AppSync and Step Functions, and you want to visualize the dd-trace and AWS X-Ray spans in one single trace.

You shared "there isn't official support for merging non-lambda X-Ray traces with DD Lambda traces". This seems to conflict with the docs - could you help me understand what I'm missing?

If I'm understanding these limitations correctly, I'd be happy to open an FR. Thanks!

[tianchu]

Sorry, to clarify, that is supported if your Lambda function also has X-Ray tracing enabled (flip the active tracing option in lambda console). This is because we only collect X-Ray traces with Lambda segments currently (we are thinking to expand that, if you submit a FR it would help the product team to prioritize). Perhaps you can try flipping the active tracing option for your lambda function and try again?

[JJSphar]

That makes sense, thanks for clarifying! I'll give enabling active tracing a try.