logstash-plugins / logstash-input-s3

Apache License 2.0
57 stars 150 forks source link

Refactor: read sincedb time once per bucket listing #233

Closed kares closed 2 years ago

kares commented 2 years ago

instead of stat/open/read/close-ing the file on every iteration ...

This is expected to improve performance in terms of File operations, which is esp. useful when using network storage.

The only (minor) downside is we read the sincedb eagerly, even if there are no objects to be processed.