Add file locking for thread safety

logstash-plugins / logstash-input-couchdb_changes

This plugin captures the _changes stream from a CouchDB instance

Apache License 2.0

27 stars 22 forks source link

Add file locking for thread safety #3

Open untergeek opened 9 years ago

untergeek commented 9 years ago

Currently nothing prevents multiple instances of Logstash from attempting to write to the same named sequence file. This is not necessarily a common occurrence, but we should be safer here by default.

colinsurprenant commented 9 years ago

Well, there are 2 cases:

multiple input plugins with the same sequence file in the same logstash instance
multiple input plugins with the same sequence file in different logstash instances

The former can be solved by adding thread safety on SequenceDB::File. The latter will have to be solved with file locking, dealing with the dreaded flock call etc.

Are any of these 2 use cases actually make any sense or both are clear misconfigurations?

untergeek commented 9 years ago

Both seem plausible to me.

I've used the less than ideal .lck file method in the past (where you make an empty file appended with .lck and check for that as the locking mechanism). Is flock really so much worse than that?

colinsurprenant commented 9 years ago

but does it make sense to have 2 plugins share the same SequenceDB::File? if not, instead of supporting concurrent access we'd just need to prevent concurrent access which could be done using "poor's man" looking with a .lock file and leveraging the fact that FS posix rename is an atomic operation to avoid startup race condition on the .lock file. Now, I am not sure about Windows...