DataDog / datadog-lambda-js

The Datadog AWS Lambda Library for Node
Apache License 2.0
107 stars 35 forks source link

CustomTraceExtractor does not work as expected #501

Closed MartinLoeper closed 5 months ago

MartinLoeper commented 5 months ago

Expected Behavior

I am trying to do custom trace propagation in the following scenario: AWS Lambda -> Event Bridge -> AWS Lambda The trace of the second lambda function should have the same trace id as the first lambda and the aws.lambda span of the first lambda should be the parent of the seconds aws.lambda span.

Actual Behavior

The datadog-lambda-js picks up the _datadog attribute which is sent to the second lambda as event payload:

{"status":"debug","message":"datadog:Extracted trace context from Custom Extractor","traceContext":{"traceId":"10134166690075585874","sampleMode":2,"source":"event","parentId":"999770304986695473"}}

It also says:

{"status":"debug","message":"datadog:Added trace context to Xray metadata","trace":{"spanContext":{"_traceId":"10134166690075585874","_spanId":"999770304986695473","_parentId":null,"_isFinished":false,"_tags":{},"_sampling":{"priority":"2"},"_baggageItems":{},"_noop":null,"_trace":{"started":[],"finished":[],"tags":{}}},"source":"event"}}

{"status":"debug","message":"datadog:Attempting to find parent for the aws.lambda span"}

However, the next log says it sets another traceId:

Inject into carrier: {"x-datadog-trace-id":"7913624874872765514","x-datadog-parent-id":"7913624874872765514","x-datadog-sampling-priority":"1"}

The following logs confirms it:

Encoding payload: [{"trace_id":"63f0beb1823f8e26","span_id":"63f0beb1823f8e26","parent_id":"0000000000000000","name":"dns.lookup","resource":"0.0.0.0","error":0,"meta":{"_dd.p.tid":"65ec478500000000","_dd.p.dm":"-0","_dd.origin":"lambda","service":"regex-parser","env":"production","version":"1.0.0","runtime-id":"ccea07aa-3139-40e6-8b4e-a587a4513c66","component":"dns","span.kind":"client","dns.hostname":"0.0.0.0","dns.address":"0.0.0.0","language":"javascript"},"metrics":{"_dd.agent_psr":1,"_dd.top_level":1,"_dd.measured":1,"process_id":16,"_sampling_priority_v1":1},"start":1709983621547005400,"duration":133043213,"links":[],"service":"regex-parser"}]

Why isn't the traceId which was obtained by traceExtractor used by the dd-trace library?

I configured the extractor as follows:

export const traceExtractor = (event) => {
    const datadog = event._datadog;

    if (!datadog) {
        return {
            sampleMode: 2,
            source: "ddtrace",
        };
    }

    const tracingContext = {
        traceId: datadog["x-datadog-trace-id"],
        sampleMode: 2,
        source: "event",
        parentId: datadog["x-datadog-parent-id"],
    };

    console.log("tracingContext", tracingContext);

    return tracingContext;
};

Is event the proper source mode in my scenario? It is hard to find any documentation on this. It looks like dd-trace just ignores the tracingContext obtained like this.

Steps to Reproduce the Problem

-

Specifications

astuyve commented 5 months ago

closing as dupe of #500