search

DataDog / datadog-lambda-js

The Datadog AWS Lambda Library for Node
Apache License 2.0
107 stars 35 forks source link

Getting 413 from Datadog when sending metrics #491

Closed aaleksandrov closed 5 months ago

aaleksandrov commented 5 months ago

Expected Behavior

We send metrics from lambda to Datadog using sendDistributionMetricWithDate (it could be 10k metrics within one invocation). We expect to see metrics on Datadog

Actual Behavior

We receive 413 message | datadog:failed attempt to send metrics to Datadog. HTTP error code: 413

I see a similar issue was raised for the python library and it was fixed by implementing compression. is it possible to do the same?

Steps to Reproduce the Problem

Send too many metrics

Specifications

Stacktrace

......:/aws/lambda/anomaly-detection-prod
2024/02/27/[$LATEST]3bcabca478e6484eba931ad8693a0dfb
2024-02-27T14:13:49.022Z    5bfee822-85f3-4b9c-a480-ca79831b8afe    ERROR   [dd.trace_id=763581508228917774 dd.span_id=4987919669838945685] {"status":"error","message":"Failed to send metrics to Datadog","name":"Error","stack":"Error: Failed to send metrics to Datadog\n    at Processor.<anonymous> (/var/task/node_modules/datadog-lambda-js/src/metrics/processor.ts:96:13)\n    at step (/var/task/node_modules/datadog-lambda-js/dist/metrics/processor.js:33:23)\n    at Object.throw (/var/task/node_modules/datadog-lambda-js/dist/metrics/processor.js:14:53)\n    at rejected (/var/task/node_modules/datadog-lambda-js/dist/metrics/processor.js:6:65)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)"}
.js:6:65)    at processTicksAndRejections (node:internal/process/task_queues:95:5)
error
astuyve commented 5 months ago

Hey @aaleksandrov, thanks for this note. We can definitely enable compression, metrics should compress well.

For anyone who may find this ticket and want to work on it, I believe we'd simply need to gzip the payload and set the Content-Encoding: deflate header, as per this pr.

Thanks!

aaleksandrov commented 5 months ago

@astuyve Could you check the related PR please? https://github.com/DataDog/datadog-lambda-js/pull/492

astuyve commented 5 months ago

This is out in v8.107.0.

Thanks again for the contribution!