DNSResolvePlugin.finish throws TypeError

markvandertol commented 9 months ago

Expected Behavior

Log requests to APM and return let request OK

Actual Behavior

After updating from 94 to 101 we start to see some requests failing with the following error:

{
  "errorType": "TypeError",
  "errorMessage": "id.toArray is not a function",
  "stack": [
    "TypeError: id.toArray is not a function",
    "    at AgentEncoder._encodeId (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:168:13)",
    "    at AgentEncoder._encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:96:12)",
    "    at AgentEncoder.encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:45:10)",
    "    at Writer._encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:39:19)",
    "    at Writer.append (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:35:10)",
    "    at AgentExporter.export (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/agent/index.js:45:18)",
    "    at SpanProcessor.process (/var/task/node_modules/dd-trace/packages/dd-trace/src/span_processor.js:50:30)",
    "    at DatadogSpan.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/opentracing/span.js:183:21)",
    "    at DNSResolvePlugin.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/plugins/tracing.js:58:22)",
    "    at DNSResolvePlugin.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/plugins/outbound.js:80:11)"
  ]
}

Steps to Reproduce the Problem

Just run requests. 1 in about 20 requests fail

Specifications

Datadog Lambda Layer version: 101
Node version: 18.x

Stacktrace

TypeError: id.toArray is not a function
    at AgentEncoder._encodeId (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:168:13)
    at AgentEncoder._encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:96:12)
    at AgentEncoder.encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:45:10)
    at Writer._encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:39:19)
    at Writer.append (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:35:10)
    at AgentExporter.export (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/agent/index.js:45:18)
    at SpanProcessor.process (/var/task/node_modules/dd-trace/packages/dd-trace/src/span_processor.js:50:30)
    at DatadogSpan.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/opentracing/span.js:183:21)
    at DNSResolvePlugin.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/plugins/tracing.js:58:22)
    at DNSResolvePlugin.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/plugins/outbound.js:80:11)

joeyzhao2018 commented 9 months ago

We just merged a PR fixing this issue. Will try to do a release today.

joeyzhao2018 commented 9 months ago

This issue is fixed in layer version 102. https://github.com/DataDog/datadog-lambda-js/releases/tag/v7.102.0

markvandertol commented 9 months ago

I'm still seeing the same error with NodeLayer version 102. These two stack traces I still see on version 102:

TypeError: id.toArray is not a function
    at AgentEncoder._encodeId (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:168:13)
    at AgentEncoder._encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:96:12)
    at AgentEncoder.encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:45:10)
    at Writer._encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:39:19)
    at Writer.append (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:35:10)
    at AgentExporter.export (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/agent/index.js:45:18)
    at SpanProcessor.process (/var/task/node_modules/dd-trace/packages/dd-trace/src/span_processor.js:50:30)
    at DatadogSpan.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/opentracing/span.js:183:21)
    at NetTCPPlugin.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/plugins/tracing.js:58:22)
    at NetTCPPlugin.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/plugins/outbound.js:80:11)

TypeError: id.toArray is not a function
    at AgentEncoder._encodeId (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:168:13)
    at AgentEncoder._encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:96:12)
    at AgentEncoder.encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:45:10)
    at Writer._encode (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:39:19)
    at Writer.append (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:35:10)
    at AgentExporter.export (/var/task/node_modules/dd-trace/packages/dd-trace/src/exporters/agent/index.js:45:18)
    at SpanProcessor.process (/var/task/node_modules/dd-trace/packages/dd-trace/src/span_processor.js:50:30)
    at DatadogSpan.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/opentracing/span.js:183:21)
    at DNSLookupPlugin.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/plugins/tracing.js:58:22)
    at DNSLookupPlugin.finish (/var/task/node_modules/dd-trace/packages/dd-trace/src/plugins/outbound.js:80:11)

I confirmed in the lambda console it uses layer version 102

benconnito commented 9 months ago

i also still see the same error in 102

{
  "errorType": "TypeError",
  "errorMessage": "id.toArray is not a function",
  "trace": [
    "TypeError: id.toArray is not a function",
    "    at AgentEncoder._encodeId (/opt/nodejs/node18/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:168:13)",
    "    at AgentEncoder._encode (/opt/nodejs/node18/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:96:12)",
    "    at AgentEncoder.encode (/opt/nodejs/node18/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:45:10)",
    "    at Writer._encode (/opt/nodejs/node18/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:39:19)",
    "    at Writer.append (/opt/nodejs/node18/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:35:10)",
    "    at AgentExporter.export (/opt/nodejs/node18/node_modules/dd-trace/packages/dd-trace/src/exporters/agent/index.js:45:18)",
    "    at SpanProcessor.process (/opt/nodejs/node18/node_modules/dd-trace/packages/dd-trace/src/span_processor.js:50:30)",
    "    at DatadogSpan.finish (/opt/nodejs/node18/node_modules/dd-trace/packages/dd-trace/src/opentracing/span.js:183:21)",
    "    at /opt/nodejs/node18/node_modules/dd-trace/packages/dd-trace/src/tracer.js:75:18",
    "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
  ]
}

joeyzhao2018 commented 9 months ago

Hi @markvandertol and @benconnito, may I ask if your lambdas have DD_MERGE_XRAY_TRACES enabled ? Thank you.

markvandertol commented 9 months ago

Hi, yes I have DD_MERGE_XRAY_TRACES enabled.

benconnito commented 9 months ago

@joeyzhao2018 i do not have it enabled

joeyzhao2018 commented 8 months ago

Hi @markvandertol and @benconnito, sorry to bother you again but I'm having a hard time reproducing the issue. I suspect this issue is still cause by some tracecontext extractor somewhere but I don't know which extractor exactly. Could you please briefly describe what upstream services are used to invoke your lambdas? And if you have set DD_TRACE_PROPAGATION_STYLE to some value specifically? Thank you in advance! 🙇

markvandertol commented 8 months ago

No, I don't specify DD_TRACE_PROPAGATION_STYLE. The lambda that fails is running as an API Gateway HTTP-API Authorizer. Maybe that helps for context.

benconnito commented 8 months ago

@joeyzhao2018 joeyzhao2018 im getting the errors (or at least most notable volume) when step functions is invoking the lambda

thnkrn commented 8 months ago

I'm also facing this error with serverless-plugin-datadog@5.53.0 and I do enable the xrayTracing

{
  "errorType": "TypeError",
  "errorMessage": "id.toArray is not a function",
  "trace": [
    "TypeError: id.toArray is not a function",
    "    at AgentEncoder._encodeId (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:168:13)",
    "    at AgentEncoder._encode (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:96:12)",
    "    at AgentEncoder.encode (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/encode/0.4.js:45:10)",
    "    at Writer._encode (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:39:19)",
    "    at Writer.append (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/exporters/common/writer.js:35:10)",
    "    at AgentExporter.export (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/exporters/agent/index.js:45:18)",
    "    at SpanProcessor.process (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/span_processor.js:50:30)",
    "    at DatadogSpan.finish (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/opentracing/span.js:183:21)",
    "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:75:18",
    "    at processTicksAndRejections (internal/process/task_queues.js:95:5)"
  ]
}

But after I'm disabling it, the step functions are working properly now

joeyzhao2018 commented 8 months ago

@junioramilson Hi, sorry to bother you again. But just want to check if your lambda has DD_MERGE_XRAY_TRACES enabled? or were you using any custom extractor? Thank you very much.

joeyzhao2018 commented 8 months ago

Please try use layer version 104. It should fix this issue. Feel free to comment here or reopen the issue if you still experience it. Thank you.

DataDog / datadog-lambda-js