lambda layer 99 introduces intermittent postgres prepared statement error

[kcveitch]

updating from layer 98 to layer 99 introduces an intermittent postgres error with prepared statements.

Error: Prepared statements must be unique - 'InsertMarketNotifications' was used for a different statement

This appears to occur when multiple invocations of a lambda are happening in rapid sequence. In this case one lambda is processing a SQS queue of a few dozen records and are posting them to a REST API Gateway which is invoking another lambda to process and insert records into postgres. The lambda being invoked from the API Gateway to insert records into postges is the one experiencing the errors and is configured with the following settings.

It appears that the first few invocations are able to make their inserts into pg without error but as more invocations come in, the error begins to occur consistently until all invocations fail in the same way.

We are using a few DD products in BETA such as serverless profiling and ASM which could be related to the issue.

reverting to nodeLayerVersion 98 corrects the issue and the error no longer occurs

Specifications

Node 16
Typescript Built using Esbuild with dd-trace/esbuild plugin
nodeLayerVersion: 99
extensionLayerVersion: 50

DD_SERVICE: !Ref AppShortName
DD_ENV: !Ref AWSEnvironment
DD_VERSION: !Ref BitbucketBuildNumber
DD_TRACE_DEBUG: false
DD_TRACE_STARTUP_LOGS: false
DD_LOGS_INJECTION: true
DD_APPSEC_ENABLED: true
DD_PROFILING_ENABLED: true
DD_SERVERLESS_APPSEC_ENABLED: true
DD_PROFILING_CODEHOTSPOTS_ENABLED: true
DD_PROFILING_ENDPOINT_COLLECTION_ENABLED: true
AWS_LAMBDA_EXEC_WRAPPER: /opt/datadog_wrapper
DD_TRACE_ENABLED: true
DD_DBM_PROPAGATION_MODE: "full"
DD_TAGS: !Sub "git.commit.sha:${BitbucketCommit},git.repository_url:https://bitbucket.org/${BitbucketRepoFullName}"

  "dependencies": {
    "axios": "^1.6.0",
    "bluebird": "^3.7.2",
    "dayjs": "^1.11.9",
    "dotenv": "^16.0.1",
    "esbuild": "^0.18.2",
    "fast-xml-parser": "^4.2.5",
    "fs": "^0.0.1-security",
    "moment": "^2.29.4",
    "node-gyp": "^9.3.1",
    "pg": "^8.11.0",
    "pg-native": "^3.0.1",
    "pg-promise": "^11.5.0",
    "uuid": "^9.0.0"
  }

Stack Trace

Error: Prepared statements must be unique - 'InsertMarketNotifications' was used for a different statement
    at wo.submit (/tmp/tmp2dqgcwz8/node_modules/pg/lib/query.js:156:14)
    at ti._pulseQueryQueue (/tmp/tmp2dqgcwz8/node_modules/pg/lib/client.js:486:45)
    at ti.query (/tmp/tmp2dqgcwz8/node_modules/pg/lib/client.js:586:10)
    at /opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/pg.js:71:28
    at exports.AsyncResource.runInAsyncScope (node:async_hooks:203:9)
    at ti.query (/opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/pg.js:55:26)
    at executor (/tmp/tmp2dqgcwz8/node_modules/pg-promise/lib/query.js:156:27)
    at E.e._execute (/tmp/tmp2dqgcwz8/node_modules/bluebird/js/release/debuggability.js:384:9)
    at E.E._resolveFromExecutor (/tmp/tmp2dqgcwz8/node_modules/bluebird/js/release/promise.js:518:18)
    at E (/tmp/tmp2dqgcwz8/node_modules/bluebird/js/release/promise.js:103:10)
    at $p (/tmp/tmp2dqgcwz8/node_modules/pg-promise/lib/promise-parser.js:30:20)
    at lT.LC (/tmp/tmp2dqgcwz8/node_modules/pg-promise/lib/query.js:145:12)
    at lT.<anonymous> (/tmp/tmp2dqgcwz8/node_modules/pg-promise/lib/query.js:277:23)
    at db (/tmp/tmp2dqgcwz8/node_modules/pg-promise/lib/database.js:333:42)
    at /opt/nodejs/node_modules/dd-trace/packages/datadog-instrumentations/src/helpers/promise.js:26:23
    at AsyncResource.runInAsyncScope (node:async_hooks:203:9)

[astuyve]

Hey, thanks for reaching out! I'm sorry you're running into an issue here.

In this case, I think this issue is best opened in dd-trace as I don't think it runs through this package specifically. Would you mind opening an issue there?

Thanks!

[kcveitch]

no problem. I debated putting it there :)

[kcveitch]

Just a note for future readers:

looks like I may have jumped the gun on this being caused by 99. let it run for 24 hours on 98 with ext 50 and still seeing the issue. looks like it might have been introduced by the agent update. Moving this ticket over to that project after I confirm that downgrading to 49 indeed resolves it.

nope was right the first time. got my deployment environments crossed

issue reposted on dd-trace-js project here: https://github.com/DataDog/dd-trace-js/issues/3793