Lambd sendDistributionMetrics - datadog:handler not initialized

DanielScharfsteinGit commented 2 months ago

Expected Behavior

Custom metrics should be added using the sendDistributionMetric function.

Actual Behavior

An error is encountered: {"status":"error","message":"datadog:handler not initialized"}.

Steps to Reproduce the Problem

I have a Lambda function that runs with the datadog Lambda Layers on Node 18.
The sendDistributionMetric is called during the runtime of the handler, after the init.
Tracer is working in the env
No webpack or any other module bundler.

Attempted Solutions

I reviewed the following issues on GitHub (https://github.com/DataDog/datadog-lambda-js/issues/298, https://github.com/DataDog/datadog-lambda-js/issues/209, https://github.com/DataDog/datadog-lambda-js/issues/369), but was unable to resolve the issue.

Specifications

Node Layer Version: 96
Extension Layer Version: 47
Node Version: 18

purple4reina commented 2 months ago

Hi @DanielScharfsteinGit, do you mind upgrading your version of datadog-lambda-js to the most recent (currently v8.108.0) and testing again? Support for your use case should be added in a more recent version.

DanielScharfsteinGit commented 2 months ago

@purple4reina Hey! Im using the newest - 8.108.0 version.

Adding that I'm not using the datadog() function from the datadog-lambda-js package but only use the layer. When adding the datadog() it is working. But I looked online and saw examples of using the metrics without the need of the datadog().

duncanista commented 2 months ago

Hey @DanielScharfsteinGit, could you share the example you are mentioning?

sendDistributionMetrics exported from this package, requires that datadog is used. Even though we allow to use it outside of the handler, usage of it is tied of our instrumentation knowing when to flush data, as datadog is never ran, our metrics processes never send that data.

DanielScharfsteinGit commented 2 months ago

Hello @duncanista,

On this page - https://docs.datadoghq.com/serverless/aws_lambda/metrics/#submit-custom-metrics, it appears that if you use the Datadog Lambda layer—which includes instrumentation and ensures tracing works—you might not need to directly invoke the datadog() function. My code snippet below, which sends custom metrics, is incorporated directly into the handler:

import { sendDistributionMetricWithDate } from 'datadog-lambda-js';

const testHandler = async (event: Event): Promise<void> => { sendDistributionMetricWithDate('example.metric', 1, new Date()); };

const handlerName = 'testHandler';

export const handler = middy(testHandler).use(logEvent(handlerName));

duncanista commented 2 months ago

@DanielScharfsteinGit,

The instructions assume that you have Serverless Intrumentation set:

Install Serverless Monitoring for AWS Lambda and ensure that you have installed the Datadog Lambda extension.

Which, depending on your instrumentation of choice, most of them directly end up using the datadog wrapper. Like changing the code handler to the Datadog one, which ends up using that function under the hood. For example, custom instrumentation requires you to do it manually.

How are you instrumenting your AWS Lambda?

duncanista commented 1 month ago

Hey,

I'm going to close this issue since there hasn't been a response in a week, and the underlying problem should be solved by instrumenting the AWS Lambda function – since the methods expect it to be eventually executed.

If there's any other questions/concerns, please feel free to re-open!

grbinho commented 4 weeks ago

I ran into a related issue with sendDistributionMetric.

For our lambda function, we are using Node.js in a Docker container with the lambda extensions.

This is the setup for the docker container

FROM shared:local as builder
WORKDIR /usr/app
COPY package.json *.ts  ./lambdas/s3-object-age-monitor/
WORKDIR lambdas/s3-object-age-monitor
RUN npm install
RUN npm run build

FROM public.ecr.aws/lambda/nodejs:20.2024.04.11.15
RUN npm install datadog-lambda-js dd-trace
COPY --from=public.ecr.aws/datadog/lambda-extension:58 /opt/. /opt/
WORKDIR ${LAMBDA_TASK_ROOT}
COPY --from=builder /usr/app/lambdas/s3-object-age-monitor/dist/* ./
ENV DD_LAMBDA_HANDLER=index.handler
ENV DD_SITE=datadoghq.com
ENV DD_LOGS_INJECTION=true
CMD ["node_modules/datadog-lambda-js/dist/handler.handler"]

And in our handler we call sendDistributionMetric as described in the documentation (without additionally wrapping our handler in index.js with datadog).

With that setup we were not getting custom metrics in data dog.

We have set DD_LOG_LEVEL=debug to try and debug and we noticed that enhanced metrics were sent, but not our custom metrics.

2024-06-18 12:22:49 UTC | DD_EXTENSION | DEBUG | Sending sketches payload : {"metric":"aws.lambda.enhanced.runtime_duration","tags":["runtime:image","region:us-east-2","architecture:x86_64", ...}

Only after we wrapped our handler in datadog did we start seeing custom metrics.

export const handler = datadog(objectAgeMonitorHandler)

If this is indeed expected behaviour, it would be very helpful if you updated documentation on this. If not, there is definitely some bug with custom metrics sending.

DataDog / datadog-lambda-js