Open StephanErb opened 3 months ago
backoff.init and backoff.max should use a higher default and leverage a jitter. There is currently no jitter in the backoff which leads to a synchronized retry across all agents.
+1 thanks for pointing this out, we should be using pseudo-random exponential backoff, not just exponential backoff.
max_retries should be reduced for metric data sets (but not logs).
Metricbeat should already default to 3 retries, while Filebeat defaults to infinite retries. Agent doesn't modify this part of the Metricbeat configuration. https://www.elastic.co/guide/en/beats/metricbeat/current/elasticsearch-output.html#_max_retries
Pinging @elastic/elastic-agent (Team:Elastic-Agent)
https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ has the modifications to the backoff algorithm we should make along with their relative performance.
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)
Describe the enhancement:
The scale preset introduced in https://github.com/elastic/kibana/issues/166870 and https://github.com/elastic/elastic-agent/issues/3797 is a great start to make agent ingestion more scalable. However, assuming one wants to ingest using thousand of agents, there is a high risk of a thundering herd problem. If Elastic starts choking the the agents reconnect and retry logic further increases the load on the cluster. This might kick the entire system over the edge.
This should be addressed with further options being tuned by the preset.
Describe a specific use case for the enhancement or feature:
The scale preset should adjust the following additional options:
backoff.init
andbackoff.max
should use a higher default and leverage a jitter. There is currently no jitter in the backoff which leads to a synchronized retry across all agents.max_retries
should be reduced for metric data sets (but not logs). It is better to focus on delivering recent data then retrying the delivery of outdated scrapes. This will act as a form of back pressure to ensure a single failure does not lead to further increased load on the server side.What is the definition of done?
Better defaults with no actions needed by users.