search

elastic / apm

Elastic Application Performance Monitoring - resources and general issue tracking for Elastic APM.
https://www.elastic.co/apm
Apache License 2.0
384 stars 114 forks source link

Elastic APM Agent Fails to Send Data: 'Read timed out, response code is -1 #879

Closed rbkuscu closed 4 months ago

rbkuscu commented 4 months ago

When the Elastic APM agent tries to send data to the APM server, it intermittently fails with the error "Read timed out, response code is -1." The agent logs also show warnings about the response body being null.

To Reproduce Steps to reproduce the behavior:

  1. Use this config: /bin/sh -c "java -Xms2G -Xmx4G -javaagent:/opt/deployment/elastic-apm-agent-1.48.1.jar -Delastic.apm.service_name=auth-service -Delastic.apm.server_url= https://elasticfleet.xxx.com/ -Delastic.apm.hostname=10.90.xxx.xx -Delastic.apm.environment=production -Delastic.apm.application_packages=com.xxx.authentication.service -Dspring.profiles.active=production -jar /opt/deployment/auth-service/auth-service.jar --server.port=8081 >> /opt/deployment/auth-service/logs/auth-service.log"

  2. Deploy the application with the above APM agent configuration.

  3. Trigger transactions that should be reported to the APM server.

  4. Observe the following error in the logs: 2024-07-18 12:51:26,873 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: null 2024-07-18 12:51:26,873 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 0 seconds (+/-10%) 2024-07-18 12:51:26,873 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Read timed out, response code is -1 2024-07-18 12:53:12,713 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: null 2024-07-18 12:53:12,713 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 0 seconds (+/-10%) 2024-07-18 12:53:12,713 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Read timed out, response code is -1

Expected behavior I expected the Elastic APM agent to successfully send data to the APM server without encountering timeouts or null response body warnings.

Debug logs 2024-07-18 12:51:26,873 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: null 2024-07-18 12:51:26,873 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 0 seconds (+/-10%) 2024-07-18 12:51:26,873 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Read timed out, response code is -1 2024-07-18 12:53:12,713 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: null 2024-07-18 12:53:12,713 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 0 seconds (+/-10%) 2024-07-18 12:53:12,713 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Read timed out, response code is -1

simitt commented 4 months ago

Hi @rbkuscu, could you please create a post in the discuss forum for this and categorize it with apm and server. Ensure to also look at the apm-server logs and include them if possible (sanitizing any potentially sensitive information first). It will allow other users to follow along with problems and discussions in one forum.

We aim for keeping the github repos for confirmed bugs and feature requests.