search

DataDog / datadog-agent

Main repository for Datadog Agent
https://docs.datadoghq.com/
Apache License 2.0
2.8k stars 1.18k forks source link

Unable to read payload context deadline exceeded #12130

Open strowk opened 2 years ago

strowk commented 2 years ago

Describe what happened: Recently (since May ~10 I think) we started to periodically get this message in logs of datadog-agent:

Error code "400 Bad Request" received while sending transaction to "https://7-35-2-app.agent.datadoghq.eu/api/v1/series?api_key=***************************": "Unable to read payload context deadline exceeded", dropping it

Now I read this that agent was sending too big a payload, which caused server to timeout reading the payload. Or maybe network was too slow. Is it some sort of wide spread issue on servers or some change in backend? Or should I adjust something in agent config? We are using latest-jmx image, so it is always the newest. These errors cause our alerts to go off and this is very noisy. I am used (a bit) to datadog-agent spamming periodically some error messages about timeouts and such, but this was not so often before and this message is relatively new. Before we maybe could have 2 errors a month, usually about timeouts between AWS and agent. Now we have this message every 2-3 days.

Describe what you expected:

Steps to reproduce the issue:

Additional environment details (Operating System, Cloud provider, etc): AWS, Fargate container, we are attaching dd-trace-java too, which I have updated in April, but it does not seem to be directly connected (errors seem to start appearing couple of weeks later).

delenamalan commented 9 months ago

We started to get this error recently very sporadically when pushing metrics to Datadog using the API. There doesn't seem to be a discernable pattern as to when this error is returned.

don41382 commented 9 months ago

Same issue here. We noticed an increase since 18th of November. In addition, the forwarder seems to have problems too:

2023-11-23 04:48:58 UTC | CORE | ERROR | (comp/forwarder/defaultforwarder/transaction/transaction.go:359 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "https://7-49-1-app.agent.datadoghq.eu/api/v1/check_run": "Unable to read payload context deadline exceeded", dropping it
nabillarahmanizhafira commented 3 months ago

do we have any update on this? because I also have similar issue with this log:

May 18 15:04:01 datadog-agent agent[14557]: 2024-05-18 15:04:01 WIB | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:350 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "https://7-43-0-app.agent.datadoghq.com/api/v2/series": "\n\x1cUnable to decomp>
May 18 15:04:20 datadog-agent agent[14557]: 2024-05-18 15:04:20 WIB | CORE | INFO | (pkg/serializer/serializer.go:413 in sendMetadata) | Sent metadata payload, size (raw/compressed): 750/357 bytes.

Thanks