search

logstash-plugins / logstash-input-s3

Apache License 2.0
57 stars 150 forks source link

Refactor the shutdown sequence of this plugin to follow the new semantic #55

Closed ph closed 8 years ago

ph commented 9 years ago

This plugin is now interruptable, this plugin require a bit more care to shutdown it down for a few reasons:

  1. It can be stuck streaming file to disk from a S3 bucket.
  2. It can be stuck reading a large file

For case 1, we cancel the streaming and delete the incomplete temp file and not update the sincedb.

For case 2, we will cancel reading the file and not update the sincedb, this will result in duplicates in the logging stream which is fine for now since the S3 plugin doesn't support saving offset yet. see #54

This PR also add a fix to non deterministic test run in a local environment by specifying a new sincedb for each run.

Fixes: #53 #42

ph commented 9 years ago

@talevy updated with your comments

talevy commented 8 years ago

LGTM

untergeek commented 8 years ago

The change to plugin from config makes the tests read much more clearly. :+1:

LGTM

ph commented 8 years ago

I'll rebase this and merge it, thanks for the reviews!

elasticsearch-bot commented 8 years ago

Merged sucessfully into master!