Closed rdner closed 2 years ago
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)
The primary use case we need to support is ensuring inputs like filebeat only update their persistent registry when events have been accepted by the output.
Filebeat's events should only be acknowledged based on the shipper's persisted index. Ideally we can create test cases proving that acknowledgements work correctly when either filebeat or the shipper restarts unexpectedly.
Describe the enhancement:
Currently, the shipper is acknowledging the event batch based on
AcceptedCount
coming from the shipper as a part of thePublishReply
.This is unsafe because
AcceptedCount
does not mean that the actual final output (e.g. Elasticsearch) acknowledged these events from the Shipper's queue, it just means that the shipper put it in its own queue for processing. So, the data can be lost if the shipper process dies.To fix this we need to use
PersistedIndex
fromPublishReply
or a separatePersistedIndex
endpoint to acknowledge the events.