Unable to use datadog-lambda-js with webpack

[raghav-fabric]

Overview

We're trying to use serverless-plugin-datadog on our service and our primary objective is to send custom metrics to datadog. We're using webpack to build our service to push it to lambdas. Before I explain further, we did find webpack had some issues with datadog-lambda-js, but we have included what ever has been mentioned here, on datadog's website for Node JS and webpacks.

Errors

When we exclude datadog-lambda-js and perform the build with serverless deployment, the function with datadog-lambda-js library being used, throws us an error as : Error: Cannot find module 'datadog-lambda-js. However, when we run the same code on local, we include datadog-lambda-js on our dev dependencies, and everything works as expected. It's just the lambda that throws errors.

Additional details

1. We have imported serverless-plugin-datadog on our code.
2. Once deployed to lambdas, we do see the datadog lambda layers.
3. If we use these layers on a manually created lambda, and use datadog-lambda-js, everything works as expected.

Our code and process

Serverless file:

plugins:
  - serverless-plugin-warmup
  - serverless-plugin-datadog
  - serverless-webpack
  - serverless-offline
  - serverless-domain-manager
  - serverless-step-functions
  - serverless-plugin-resource-tagging
  - serverless-dynamo-stream-plugin

custom:
  datadog:
    addExtension: true
    addLayers: true
    apiKey: <my-dd-key>
    flushMetricsToLogs: false
    enabled: true
    injectLogContext: true
    logLevel: debug
    enableDDTracing: false
  webpack:
    webpackConfig: /webpack.config.js
    includeModules:
      forceExclude: 
        - datadog-lambda-js
        - dd-trace
    packagerOptions: 
      scripts:
        - rm -rf node_modules/datadog-lambda-js node_modules/dd-trace
    packager: npm

Webpack file (excludes value): externals: [nodeExternals(), "dd-trace", "datadog-lambda-js"],

Datadog file

const { datadog, sendDistributionMetric, sendDistributionMetricWithDate } = require('datadog-lambda-js');

exports.hello = async function hello(event, context) {
    console.log("Running metrics")
    sendDistributionMetricWithDate(
      "<my-metric-name>",       // Metric name
      1,                                         // Metric value
      new Date(Date.now()),                            
      "tag1:val1",
      "tag2:val2",  // Associated tags

    );
    console.log("Closing metrics")
    return {
      statusCode: 200,
      body: "hello, dog!",
    };
  }

Specifications

• Serverless Framework version: 2.62.0
• Datadog Serverless Plugin version: "serverless-plugin-datadog": "^5.1.1",
• Lambda function runtime (Python 3.7, Node 10, etc.): Node 14.x

References:

1. Sending Custom metrics with serverless to datadog
2. StackOverflow Question

Please let me know If I'm missing something here, but we've been trying from sometime now and still facing the same issues where datadog-lambda-js is not being recognised on our function even when we have the lambda layers.

[astuyve]

Hi @raghav-fabric - thanks for reaching out. I'll reproduce with your example and try to be of assistance.

[raghav-fabric]

Hi @raghav-fabric - thanks for reaching out. I'll reproduce with your example and try to be of assistance.

Thanks @astuyve Let me know if you want me to share any additional code from my end. Really appreciate your help on this. 👍

[DarcyRaynerDD]

Generally, if you are using the serverless-webpack plugin, we should be force excluding dd-lambda-js from your dependencies when you bundle and deploy. So it might potentially be a bug or edge case

[DarcyRaynerDD]

Is dd-lambda-js/dd-trace-js installed locally in your package.json while developing?

[raghav-fabric]

@DarcyRaynerDD Thanks for checking on this. but datadog-lambda-js and dd-trace-js both are not included in my package.json. They're removed while deploying. I've also removed them using webpack scripts (as mentioned here : https://docs.datadoghq.com/serverless/guide/serverless_tracing_and_webpack/)

[astuyve]

Hi @raghav-fabric - one thing to check is placing serverless-plugin-datadog at the end of the list of your plugins, this will run our plugin last, ensuring webpack has finished before the datadog handler redirection is applied.

Given that change, I can't reproduce your issue with the following minimal example: src/handler.js

const { datadog, sendDistributionMetric, sendDistributionMetricWithDate } = require('datadog-lambda-js');

exports.hello = async function hello(event, context) {
    console.log("Running metrics")
    sendDistributionMetricWithDate(
      "<my-metric-name>",       // Metric name
      1,                                         // Metric value
      new Date(Date.now()),
      "tag1:val1",
      "tag2:val2",  // Associated tags

    );
    console.log("Closing metrics")
    return {
      statusCode: 200,
      body: "hello, dog!",
    };
  }

webpack.config.js

var nodeExternals = require("webpack-node-externals");

module.exports = {
  entry: './src/handler.js',
  // use webpack-node-externals to exclude all node dependencies.
  // You can manually set the externals too.
  // output: {
  //   path: __dirname + '/src',
  //   filename: '[name].js'
  // },
  externals: [nodeExternals(), "dd-trace", "datadog-lambda-js"],
};

serverless.yml

service: js-webpack
frameworkVersion: "3"

provider:
  name: aws
  runtime: nodejs14.x

plugins:
  - serverless-webpack
  - serverless-plugin-datadog

custom:
  datadog:
    addExtension: true
    addLayers: true
    apiKey: <my API Key>
    logLevel: debug
  webpack:
    webpackConfig: /webpack.config.js
    includeModules:
      forceExclude:
        - datadog-lambda-js
        - dd-trace
    packagerOptions:
      scripts:
        - rm -rf node_modules/datadog-lambda-js node_modules/dd-trace
    packager: npm

functions:
  hello:
    handler: src/handler.hello
    events:
      - httpApi:
          path: /hello
          method: get

This results in a 200ok response, metrics being delivered, traces collected, and things generally working:

If the datadog node layer is present, I'm unclear how the require would fail in this case.

[raghav-fabric]

Thanks for this @astuyve . Let me deploy with serverless plugin datadog on the end. And yes, I do get the layers on my lambdas. Not only this, but if I import these layers by ARN on a different lambda (created manually) and set the datadog custom metric code on that lambda, and then fire a test on it, it works as expected and used the library from the layers. I'm still confused if webpack is blocking something here.

While I try this with my deployment, I'll share a few more things that may be of use. Let me know if these change anything for our process.

Details included on serverless:

package:
  individually: false
  excludeDevDependencies: true

Webpack code

const path = require('path')
const slsw = require('serverless-webpack')
const nodeExternals = require('webpack-node-externals')
const HtmlWebpackPlugin = require('html-webpack-plugin')

module.exports = {
  entry: slsw.lib.entries,
  target: 'node',
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  optimization: {
    // We no not want to minimize our code.
    minimize: !slsw.lib.webpack.isLocal
  },
  performance: {
    // Turn off size warnings for entry points
    hints: false
  },
  devtool: 'source-map',
  externals: [nodeExternals(), "dd-trace", "datadog-lambda-js"],
  resolve: {
    alias: {
      app: path.join(process.cwd(), 'app')
    }
  },
  output: {
    libraryTarget: 'commonjs2',
    path: path.join(__dirname, '.webpack'),
    filename: '[name].js',
    sourceMapFilename: '[file].map'
  },
  module: {
    rules: [
      {
        test: /\.html$/i,
        loader: 'html-loader'
      },
      {
        test: /\.yaml$/i,
        type: 'json',
        loader: 'yaml-loader'
      }
    ]
  },
  plugins: [new HtmlWebpackPlugin()]
}

[raghav-fabric]

Update: It's still throwing the same error.

[astuyve]

Okay, I'll try your webpack config. Although I don't entirely suspect webpack, could you share more of your serverless.yml file? Specifically the provider block, other plugin configuration information, and a snippet of an example function?

Please redact any secrets, of course.

Thanks!

[raghav-fabric]

Sure @astuyve (updated with function and handler)

Serverless file details:

frameworkVersion: ">=1.51"
provider:
  name: aws
  endpointType: regional
  versionFunctions: false
  runtime: nodejs14.x
  stage: ${opt:stage,'local'}
  region: ${opt:region, 'us-east-1'} # our default region
  memorySize: ${self:custom.memorySize.${self:provider.stage}, ${self:custom.memorySize.env}}
  logRetentionInDays: 2
  apiName: ${self:provider.stage}-${self:service.name} 
  apiKeys:
    - ${self:provider.apiName}
  vpc:
    securityGroupIds:
      - ${env:SECURITY_GROUP_ID, ''}
    subnetIds:
      - ${env:SUBNET_ID, ''}
  usagePlan:
    quota:
      limit: 864000000
      offset: 0
      period: DAY
    throttle:
      burstLimit: 1000
      rateLimit: 1000
  environment:
   # ... includes env variables
  stackTags:
   # ... internal stack tags
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:DescribeTable
        - dynamodb:Query
        - dynamodb:Scan
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
        - dynamodb:CreateTable
        - dynamodb:ListStreams
        - dynamodb:GetShardIterator
        - dynamodb:GetRecords
        - dynamodb:DescribeStream
        - es:ESHttpPost
        - es:ESHttpPut
        - sqs:ReceiveMessage
        - sqs:GetQueueAttributes
        - sqs:DeleteMessage
        - logs:CreateLogGroup
        - logs:CreateLogStream
        - logs:PutLogEvents
        - kinesis:DescribeStreamSummary
        - kinesis:ListShards
        - kinesis:GetShardIterator
        - kinesis:GetRecords 
        - kinesis:SubscribeToShard
        - "sns:*"
      Resource: "*"
    - Effect: Allow
      Action:
        - "s3:*"
      Resource: "*"

functions:
  datadog-push-metrics:
    name: ${self:provider.apiName}-datadog-push-metrics
    handler: app/routes/datadog/push-metrics.handler

Route:

const MetricUpdater = require('../../modules/datadog-metric/index')
const { datadog, sendDistributionMetric, sendDistributionMetricWithDate } = require('datadog-lambda-js');

exports.handler = datadog(
    async (event, context) => {
  return MetricUpdater.hello(event, context)
}
)

[astuyve]

Thanks for this @raghav-fabric - I'll attempt this with a few key changes.

I should point out that the supplied IAM permissions are extremely broad. They allow the lambda functions in this template to do things I suspect you don't intend, including the ability to delete all objects from all s3 buckets, as well as delete the buckets themselves. I'd recommend reviewing them, and if you're new to Serverless, it may help to check out this guide. I hope this is helpful!

Thanks again!

[raghav-fabric]

Thanks for this @raghav-fabric - I'll attempt this with a few key changes.

I should point out that the supplied IAM permissions are extremely broad. They allow the lambda functions in this template to do things I suspect you don't intend, including the ability to delete all objects from all s3 buckets, as well as delete the buckets themselves. I'd recommend reviewing them, and if you're new to Serverless, it may help to check out this guide. I hope this is helpful!

Thanks again!

Hi @astuyve Thanks for sharing this. Really appreciate the suggestion, however, we have multiple services that we're working with on the same project. These permissions are something we've added overtime for use (for the services that we're building). Most permissions used above are actually being used.

[astuyve]

@raghav-fabric - I haven't had any luck reproducing this with your example. I think the best solution to move forward is to open a support ticket so that you could provide a .zip file which we could test.

Could you also include a stack trace? I'd like to know if the error is being thrown from the logic in your handler which is sending a custom metric, or if it's from somewhere else.

Thanks!

[raghav-fabric]

Hi @astuyve
Thanks for helping out on this. Let me give it a try again. If this does now work, I'll raise a support ticket. Really appreciate your help. 👍

[raghav-fabric]

@astuyve I was able to solve this by changing the NODE_PATH. We had the NODE_PATH set to "." (root) earlier, we tried to change and set it to /opt/nodejs/node_modules and it fixed the issue.

Thank you for helping me out on this. Cheers. 👏

[astuyve]

Thank you so much for the followup comment @raghav-fabric. I'm sure it will help another user down the road!