Restructure for better large bucket support (Fixes #138, #128. #100, #80, #54, and #14)

[eherot]

(This is a reworking of the now defunct #84)

What this changes

• Use a synchronous queue to handle the processing of events in parallel
• Starts sending events as soon as they are ingested rather than trying to scan the entire bucket first
• Switch to aws-sdk v3
• Add a lot of logging
• Enable the use of the SDK's start_after parameter call to fetch only new events (useful in cases where objects are stored in alphabetical order by time, such as S3 access logs)
• Limit the batch size of the S3 request

May/Should fix a number of open issues either directly or by virtue of replacing the problem code entirely

• 213

• 172

• 191

• 226

• 236

• 239

• 138

• 128

• 118

• 100

• 80

• 54

• 47 — Probably somewhat, anyway. This input is inherently request-heavy when the bucket is very large because there is no way to download large numbers of S3 files in a single request.

• 14

Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/

[roaksoax]

Hi @eherot

The Logstash S3 input is now been shipped as part of https://github.com/logstash-plugins/logstash-integration-aws , rather than the individual plugin. Not sure if you want to re-target your PR against it?

[eherot]

Thanks for the heads up. Let me see how much work that's going to be...