search

elastic / elastic-agent-shipper

Data shipper for the Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
9 stars 19 forks source link

Explore an autoscaling Elasticsearch output based on the APM Server autoscaling work #175

Open cmacknz opened 1 year ago

cmacknz commented 1 year ago

APM Server now has an autoscaling Elasticsearch output (specialized to the APM data model). We should explore porting their implementation into the shipper with the goal of eliminating the need to manually tune the number of workers and the output batch size.

The initial APM autoscaling implementation was done in https://github.com/elastic/apm-server/pull/9393 and linked to the following tracking issues:

When we begin this work we should sync with the APM Server team to come up to speed on the latest state of their autoscaling implementation.

pierrehilbert commented 1 year ago

@cmacknz & @leehinman: what would be the expected outcome of this one?

cmacknz commented 1 year ago

A proof of concept of an autoscaling Elasticsearch output based on the work the APM team has done.

The ideal outcome is a demo showing that more Elasticsearch output workers are created as the rate of shipper data ingestion increases, with the number of workers then decreasing as the data ingestion rate decreases.

cmacknz commented 1 year ago

The Elasticsearch output used by APM server has been moved into a separate package: https://github.com/elastic/go-docappender

We should explore if we can just reuse this directly.