Synchronized Log Rotation and Invariant

josephlewis42 commented 6 years ago

This PR abstracts away the details of log rotation and file management from the main google_cloud_storage.rb file.

It breaks duties into three components:

LogStash::Outputs::GoogleCloudStorage - the base plugin, responsible for instantiation, event conversion (to plain text or JSON), and submitting rotated out files to the GCS client
LogStash::Outputs::LogFileFactory (and friends) - responsible for adapting raw file descriptors to something the application can use.
LogStash::Outputs::LogRotate - responsible for managing the lifecycle of log files, writing messages to them and notifying the plugin when they're ready to be uploaded.

LogRotate keeps a strict invariant that it will always have exactly one log file open per plugin instance. This will fix #2, and fix #19.

rosbo commented 6 years ago

Question on the general approach, if you use multiple pipelines, does it share the same running process of this plugin? If that's the case, should the invariant be "exactly one log file open per pipeline"?

josephlewis42 commented 6 years ago

@rosbo you're correct, it'll be one open file per plugin instance.

logstash-plugins / logstash-output-google_cloud_storage

Synchronized Log Rotation and Invariant #31