Lambda throws 'Maximum call stack size exceeded' error when using Datadog Lambda Extension

jessgoldq4 commented 2 years ago

Expected Behavior

Lambda code functions as expected.
Lambda output logs to datadog as expected.

Actual Behavior

❌ Lambda throws Maximum call stack size exceeded error upon invocation.
✅ Lambda output logs to datadog as expected.

Steps to Reproduce the Problem

install v5.0.0 of serverless-plugin-datadog
update serverless.yml:

plugins:
  - serverless-plugin-datadog
  - serverless-webpack
  - serverless-offline-local-authorizers-plugin
  - serverless-offline

custom:   
  datadog:
    site: datadoghq.com
    apiKey: ${env:DATADOG_API_KEY, ''}

run sls deploy

Specifications

Serverless Framework version: 3.7.1
Datadog Serverless Plugin version: 5.0.0
Lambda function runtime: Node 14.19.0 | highest supported version by aws

Stacktrace

{
    "errorType": "RangeError",
    "errorMessage": "Maximum call stack size exceeded",
    "stack": [
        "RangeError: Maximum call stack size exceeded",
        "    at emitInitNative (internal/async_hooks.js:210:5)",
        "    at emitInitScript (internal/async_hooks.js:493:3)",
        "    at process.nextTick (internal/process/task_queues.js:133:5)",
        "    at Channel.publish (diagnostics_channel.js:58:17)",
        "    at ServerResponse.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/http/server.js:33:24)",
        "    at onFinish (_http_outgoing.js:792:10)",
        "    at Object.write (/var/task/node_modules/@vendia/serverless-express/src/response.js:124:11)",
        "    at ServerResponse._writeRaw (_http_outgoing.js:351:17)",
        "    at ServerResponse._send (_http_outgoing.js:327:15)",
        "    at ServerResponse.end (_http_outgoing.js:849:10)"
    ]
}

Environment

Operation system: AWS
Node.js version: 14.19.0
Tracer version: Whatever serverless-plugin-datadog": "^5.0.0 installs
Agent version: Whatever serverless-plugin-datadog": "^5.0.0 installs
Relevant library versions:

  "dependencies": {
    "@apollo/subgraph": "^0.3.1",
    "apollo-server-core": "^3.6.3",
    "apollo-server-lambda": "^3.6.3",
    "graphql": "^16.3.0",
    "joi": "^17.5.0",
    "knex": "^0.95.14",
    "snowflake-sdk": "^1.6.8"
  },
  "devDependencies": {
    "@graphql-tools/webpack-loader": "^6.6.2",
    "@types/jest": "^27.0.3",
    "@typescript-eslint/eslint-plugin": "^5.6.0",
    "@typescript-eslint/parser": "^5.6.0",
    "eslint": "^8.4.1",
    "eslint-config-standard": "^16.0.3",
    "eslint-plugin-import": "^2.25.3",
    "eslint-plugin-node": "^11.1.0",
    "eslint-plugin-promise": "^5.2.0",
    "eslint-plugin-security": "^1.4.0",
    "eslint-plugin-sonarjs": "^0.11.0",
    "eslint-plugin-standard": "^5.0.0",
    "jest": "27.3.1",
    "jest-cli": "27.3.1",
    "jest-when": "^3.4.2",
    "serverless": "^3.7.1",
    "serverless-offline": "^8.5.0",
    "serverless-offline-local-authorizers-plugin": "^1.2.0",
    "serverless-webpack": "5.6.0",
    "ts-jest": "^27.0.7",
    "ts-loader": "^9.2.6",
    "ts-node": "^10.4.0",
    "typescript": "^4.5.2",
    "webpack": "^5.65.0",
    "webpack-node-externals": "^3.0.0"
  }

bengl commented 2 years ago

Hi @jessgoldq4, thanks for reporting this! Can you try running this with Error.stackTraceLimit = 100 to capture more of the stack trace? That way, we can see if there's any infinite recursion going on, and what can be done to resolve it.

nickabb commented 2 years ago

@bengl I am seeing this same issue on a fresh integration via the dd cli. datadog-ci lambda instrument -i -> Follow the prompts, lambda then begins throwing this error.

How can I configure with Error.stackTraceLimit = 100 to get you the information you need?

jessgoldq4 commented 2 years ago

Hi @bengl. Here's an updated stack trace

{
    "errorType": "RangeError",
    "errorMessage": "Maximum call stack size exceeded",
    "stack": [
        "RangeError: Maximum call stack size exceeded",
        "    at emitInitNative (internal/async_hooks.js:210:5)",
        "    at emitInitScript (internal/async_hooks.js:493:3)",
        "    at process.nextTick (internal/process/task_queues.js:133:5)",
        "    at Channel.publish (diagnostics_channel.js:58:17)",
        "    at ServerResponse.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/http/server.js:33:24)",
        "    at onFinish (_http_outgoing.js:792:10)",
        "    at Object.write (/var/task/node_modules/@vendia/serverless-express/src/response.js:124:11)",
        "    at ServerResponse._writeRaw (_http_outgoing.js:351:17)",
        "    at ServerResponse._send (_http_outgoing.js:327:15)",
        "    at ServerResponse.end (_http_outgoing.js:849:10)",
        "    at ServerResponse.<anonymous> (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31)",
        "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31",
        "    at /opt/nodejs/node_modules/dd-trace/packages/datadog-plugin-http/src/server.js:60:70",
        "    at Subscription._handler (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/plugin.js:12:9)",
        "    at Channel.publish (diagnostics_channel.js:56:9)",
        "    at ServerResponse.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/http/server.js:33:24)",
        "    at onFinish (_http_outgoing.js:792:10)",
        "    at Object.write (/var/task/node_modules/@vendia/serverless-express/src/response.js:124:11)",
        "    at ServerResponse._writeRaw (_http_outgoing.js:351:17)",
        "    at ServerResponse._send (_http_outgoing.js:327:15)",
        "    at ServerResponse.end (_http_outgoing.js:849:10)",
        "    at ServerResponse.<anonymous> (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31)",
        "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31",
        "    at /opt/nodejs/node_modules/dd-trace/packages/datadog-plugin-http/src/server.js:60:70",
        "    at Subscription._handler (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/plugin.js:12:9)",
        "    at Channel.publish (diagnostics_channel.js:56:9)",
        "    at ServerResponse.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/http/server.js:33:24)",
        "    at onFinish (_http_outgoing.js:792:10)",
        "    at Object.write (/var/task/node_modules/@vendia/serverless-express/src/response.js:124:11)",
        "    at ServerResponse._writeRaw (_http_outgoing.js:351:17)",
        "    at ServerResponse._send (_http_outgoing.js:327:15)",
        "    at ServerResponse.end (_http_outgoing.js:849:10)",
        "    at ServerResponse.<anonymous> (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31)",
        "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31",
        "    at /opt/nodejs/node_modules/dd-trace/packages/datadog-plugin-http/src/server.js:60:70",
        "    at Subscription._handler (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/plugin.js:12:9)",
        "    at Channel.publish (diagnostics_channel.js:56:9)",
        "    at ServerResponse.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/http/server.js:33:24)",
        "    at onFinish (_http_outgoing.js:792:10)",
        "    at Object.write (/var/task/node_modules/@vendia/serverless-express/src/response.js:124:11)",
        "    at ServerResponse._writeRaw (_http_outgoing.js:351:17)",
        "    at ServerResponse._send (_http_outgoing.js:327:15)",
        "    at ServerResponse.end (_http_outgoing.js:849:10)",
        "    at ServerResponse.<anonymous> (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31)",
        "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31",
        "    at /opt/nodejs/node_modules/dd-trace/packages/datadog-plugin-http/src/server.js:60:70",
        "    at Subscription._handler (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/plugin.js:12:9)",
        "    at Channel.publish (diagnostics_channel.js:56:9)",
        "    at ServerResponse.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/http/server.js:33:24)",
        "    at onFinish (_http_outgoing.js:792:10)",
        "    at Object.write (/var/task/node_modules/@vendia/serverless-express/src/response.js:124:11)",
        "    at ServerResponse._writeRaw (_http_outgoing.js:351:17)",
        "    at ServerResponse._send (_http_outgoing.js:327:15)",
        "    at ServerResponse.end (_http_outgoing.js:849:10)",
        "    at ServerResponse.<anonymous> (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31)",
        "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31",
        "    at /opt/nodejs/node_modules/dd-trace/packages/datadog-plugin-http/src/server.js:60:70",
        "    at Subscription._handler (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/plugin.js:12:9)",
        "    at Channel.publish (diagnostics_channel.js:56:9)",
        "    at ServerResponse.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/http/server.js:33:24)",
        "    at onFinish (_http_outgoing.js:792:10)",
        "    at Object.write (/var/task/node_modules/@vendia/serverless-express/src/response.js:124:11)",
        "    at ServerResponse._writeRaw (_http_outgoing.js:351:17)",
        "    at ServerResponse._send (_http_outgoing.js:327:15)",
        "    at ServerResponse.end (_http_outgoing.js:849:10)",
        "    at ServerResponse.<anonymous> (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31)",
        "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31",
        "    at /opt/nodejs/node_modules/dd-trace/packages/datadog-plugin-http/src/server.js:60:70",
        "    at Subscription._handler (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/plugin.js:12:9)",
        "    at Channel.publish (diagnostics_channel.js:56:9)",
        "    at ServerResponse.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/http/server.js:33:24)",
        "    at onFinish (_http_outgoing.js:792:10)",
        "    at Object.write (/var/task/node_modules/@vendia/serverless-express/src/response.js:124:11)",
        "    at ServerResponse._writeRaw (_http_outgoing.js:351:17)",
        "    at ServerResponse._send (_http_outgoing.js:327:15)",
        "    at ServerResponse.end (_http_outgoing.js:849:10)",
        "    at ServerResponse.<anonymous> (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31)",
        "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31",
        "    at /opt/nodejs/node_modules/dd-trace/packages/datadog-plugin-http/src/server.js:60:70",
        "    at Subscription._handler (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/plugin.js:12:9)",
        "    at Channel.publish (diagnostics_channel.js:56:9)",
        "    at ServerResponse.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/http/server.js:33:24)",
        "    at onFinish (_http_outgoing.js:792:10)",
        "    at Object.write (/var/task/node_modules/@vendia/serverless-express/src/response.js:124:11)",
        "    at ServerResponse._writeRaw (_http_outgoing.js:351:17)",
        "    at ServerResponse._send (_http_outgoing.js:327:15)",
        "    at ServerResponse.end (_http_outgoing.js:849:10)",
        "    at ServerResponse.<anonymous> (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31)",
        "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31",
        "    at /opt/nodejs/node_modules/dd-trace/packages/datadog-plugin-http/src/server.js:60:70",
        "    at Subscription._handler (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/plugin.js:12:9)",
        "    at Channel.publish (diagnostics_channel.js:56:9)",
        "    at ServerResponse.emit (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/http/server.js:33:24)",
        "    at onFinish (_http_outgoing.js:792:10)",
        "    at Object.write (/var/task/node_modules/@vendia/serverless-express/src/response.js:124:11)",
        "    at ServerResponse._writeRaw (_http_outgoing.js:351:17)",
        "    at ServerResponse._send (_http_outgoing.js:327:15)",
        "    at ServerResponse.end (_http_outgoing.js:849:10)",
        "    at ServerResponse.<anonymous> (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31)",
        "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/plugins/util/web.js:302:31"
    ]
}

bengl commented 2 years ago

Hey folks! We've been unable to reproduce this on our (Datadog) end so far. Can anyone provide a small lambda repo that reproduces this issue?

jessgoldq4 commented 2 years ago

import { ApolloServer } from 'apollo-server-lambda'
import { ApolloServerPluginLandingPageGraphQLPlayground, gql } from 'apollo-server-core'
import { buildSubgraphSchema } from '@apollo/subgraph'
import { resolvers } from './resolvers'
import typeDefs from './schema.graphql'

const server = new ApolloServer({
  schema: buildSubgraphSchema([{ typeDefs, resolvers }]),
  introspection: true,
  plugins: [ApolloServerPluginLandingPageGraphQLPlayground()]
})

export const graphql = server.createHandler()

import { SampleController } from '../../controllers'
import { resolveObj } from '../../utils'

const sampleController = new SampleController()

export const resolvers = {
  Query: {
    sampleNode: resolveObj
  },
  SampleResult: {
    items: sampleController.get,
    count: sampleController.getCount
  }
}

https://www.apollographql.com/docs/apollo-server/deployment/lambda https://www.apollographql.com/docs/federation/api/apollo-subgraph

@bengl

astuyve commented 2 years ago

Hi @jessgoldq4 - can you try this again with datadog plugin v5.1.0, and nodejs layer version 78?

jessgoldq4 commented 2 years ago

@astuyve It seems to work with that version. However, I'm unable to see logs, only traces. I've explicitly set them to true, despite them being true by default, just in case... but no luck

astuyve commented 2 years ago

Thanks @jessgoldq4 - and you can confirm that switching back to 77 fixes log and trace correlation? If that's the case, let's open a new issue here

jessgoldq4 commented 2 years ago

@astuyve I can confirm that it was the same with the older version. There were no logs either. Do I need to set up a datadog forwarder? I thought the new extension deprecated the need for that though

Also I've noticed that my function keeps getting removed from datadog for some reason. After a few hours, when I navigate infrastructure -> serverless, it's just magically gone

astuyve commented 2 years ago

Hi @jessgoldq4 - no, the datadog lambda extension can forward logs. The easiest way to verify this is happening is to visit the Logs tab in Datadog and search for your service.

The datadog extension should connect logs and traces automatically. Can you verify that the lambda.request_id tag is set on your function spans? If that's not happening, definitely open a ticket with us here.

Regarding services disappearing, Datadog's serverless APM view shows services with function invocations in the selected timeframe. The default timeframe is 15 minutes, so if your service hasn't been invoked in a while, that's likely the culprit. If that's not the issue, I'd suggest opening a support ticket, since the app side triages issues via that manner (vs github).

Based on your feedback, I feel that this issue (#2045) can be closed. If there are further serverless issues you're running into with Node, please feel free to open a ticket here, where I will be able to assist you further.

Thanks again for all your help with this issue!

jessgoldq4 commented 2 years ago

@astuyve I have resolved the issue regarding the logs. I believe it was a combination of not setting the logLevel param in serverless.yml, and the DD AWS Lambda integration IAM role missing some required permissions. Thanks for the support!

astuyve commented 2 years ago

@jessgoldq4 Great! I'm glad I could help.

DataDog / dd-trace-js