search

logstash-plugins / logstash-input-s3

Apache License 2.0
57 stars 150 forks source link

Define cut-off time in list_new_files to prevent recently-modified files being missed #189

Closed lsutcliffe-dbg closed 3 years ago

lsutcliffe-dbg commented 4 years ago

Fix for the bug detailed in https://github.com/logstash-plugins/logstash-input-s3/issues/57

When there are multiple files created with the same timestamp, it is possible that some of them are added to S3 after list_new_files has already run. The sincedb is then updated with that timestamp, so the newly-added files are not picked up on the next run.

With this change, any files modified within the last 2 seconds will not be considered and a debug log is written to say the object was modified too recently.

kaisecheng commented 3 years ago

Thank you for submitting the PR. The new release v3.6.0 fixed the problem, hence close this