search

logstash-plugins / logstash-input-s3

Apache License 2.0
57 stars 150 forks source link

added new_files_only functionality #182

Closed kpolley closed 5 years ago

kpolley commented 5 years ago

Simple functionality that, when triggered, will only ingest data in the bucket that is new or modified

elasticcla commented 5 years ago

Hi @kpolley, we have found your signature in our records, but it seems like you have signed with a different e-mail than the one used in your Git commit. Can you please add both of these e-mails into your Github profile (they can be hidden), so we can match your e-mails to your Github profile?

kpolley commented 5 years ago

@robbavey

Thanks for the feedback! I see now that list_new_files call does not change any state and my tests were inaccurate.

The goal is to skip over existing files every time logstash starts. As an example usecase, we'd like to ingest CloudTrail logs that are produced today and in the future. We do not want to ingest all of the logs.

I actually am running PR #178 in my test environment with some success (so far).