Fails to upload files when logstash exits

[wfhartford]

In situations where Logstash exits it is quite likely that the plugin will leave some files on disk, not uploaded to GCS. This is especially likely when large amounts of data is exported over a short execution.

My use case is exporting data from Elasticsearch to GCS using the logstash-input-elasticsearch plugin. This plugin will pull data out of Elasticsearch based on a configurable query. When all data from that query has been processed, Logstash will exit. In this case, it is almost certain that a significant amount of data will not be uploaded to GCS.

A related issue is covered in the Improvements TODO list in the header comment:

There's no recover method, so if logstash/plugin crashes, files may not be uploaded to GCS.

That description trivialities the issue by assuming that it only applies to crash situations, when in fact it applies any time Logstash exits including a user initiated shutdown and a successful completion. The export use case describe above transforms this from a potential minor problem to a fatal flaw.

[josephlewis42]

I've added the ability to do this for the events in the queue as well as part of #21, once that's merged this should be closable.

[josephlewis42]

Alright, this now works for both async and synchronous uploads. Synchronous uploads are still fully supported if you prefer to go that route.