elastic / elastic-agent-shipper

Data shipper for the Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
9 stars 19 forks source link

Move batch assembly / acknowledgment out of the Beats queue #186

Open faec opened 1 year ago

faec commented 1 year ago

In Beats, vending of events from the queue and acknowledgment of those events by the outputs is done in "batches," arrays of consecutive events that are sent together over the network, and are removed from the queue as a single unit when complete.

In the agent shipper, outputs no longer use the batch abstraction -- the boundary of a network "batch" can happen anywhere based on the configuration of the library (for example, go-elasticsearch can send batches based on a configured byte limit). Acknowledgments may be received per-event and need to be propagated back to the queue so completed messages can be freed from memory/disk.

Currently this mismatch between the shipper and the beats queue is handled by wrapping fixed-size queue batches in a helper object that listens to callbacks and tracks how many events from that batch are still outstanding. This decouples the go-elasticsearch workers from the batch boundaries in the originating queue. Instead, we need the core queue API to support reading single events, so we should move the assembly of those events into batches into an auxiliary helper that can preserve the same behavior in Beats while allowing the shipper to ignore the batch interface and propagate event acknowledgments directly.

pierrehilbert commented 1 year ago

@cmacknz / @leehinman could we close this one with the new shipper plan?

cmacknz commented 1 year ago

We need this for the new go-elasticsearch output, which applies to both plans.