When Filebeat is running with autodiscover enabled it can happen that a pod is terminated, autodiscover signals to terminate the input, but the files have not been read until EOF. Terminating the input on this state will lead to the file not being fully ingested.
There are at least a couple of common cases where this happens:
Short lived pods
High load environments where Filebeat might experience back pressure from the output
Currently the best way to avoid data loss on those situations is to set close.on_state_change.removed: false in the input configuration and cleanup_timeout for the autodiscover configuration to a value that will give Filebeat enough time to ingest all data.
When Filebeat is running with autodiscover enabled it can happen that a pod is terminated, autodiscover signals to terminate the input, but the files have not been read until EOF. Terminating the input on this state will lead to the file not being fully ingested.
There are at least a couple of common cases where this happens:
Currently the best way to avoid data loss on those situations is to set
close.on_state_change.removed: false
in the input configuration andcleanup_timeout
for the autodiscover configuration to a value that will give Filebeat enough time to ingest all data.