cherweg / logstash-input-s3-sns-sqs

logstash input downloading files from s3 Bucket by OjectKey from SNS/SQS
Other
29 stars 35 forks source link

S3 Objects with Versioning ? #64

Open hinchliff opened 3 years ago

hinchliff commented 3 years ago

If the S3 bucket has Versioning enabled, will this plugin use the versionId information provided when pulling the Object out of S3?

I'm looking at a use-case where files are being over-written / updated, and the usual All object create events are not sufficient. Enabling versioning in the bucket would solve this: https://forums.aws.amazon.com/thread.jspa?messageID=795363

I guess by default, pulling the object from S3 will pull the latest version? Which may work in most cases.

But if the object was overwritten multiple times in a short time period, this plugin would receive multiple notifications, and could end up pulling the same "latest" object multiple times, instead of pulling some number of different versions?