[Filestream] Filestream add flag to only terminate the input after reaching EOF on all files

When Filebeat is running with autodiscover enabled it can happen that a pod is terminated, autodiscover signals to terminate the input, but the files have not been read until EOF. Terminating the input on this state will lead to the file not being fully ingested.

There are at least a couple of common cases where this happens:

Short lived pods
High load environments where Filebeat might experience back pressure from the output

Currently the best way to avoid data loss on those situations is to set close.on_state_change.removed: false in the input configuration and cleanup_timeout for the autodiscover configuration to a value that will give Filebeat enough time to ingest all data.

elastic / beats

[Filestream] Filestream add flag to only terminate the input after reaching EOF on all files #40447