logstash-plugins / logstash-output-google_cloud_storage

Apache License 2.0
9 stars 26 forks source link

Adding "prefix" to bucket silently drops data #45

Open idsvandermolen opened 4 years ago

idsvandermolen commented 4 years ago

We would like to be able to use prefixes (similiar to logstash-s3-output plugin), but when you add a "prefix" to the bucket field, this plugin silently drops data. Config like:

google_cloud_storage {
  bucket => "my-logs/test"
  ...
}

You see a message like: [2020-02-26T15:46:26,987][INFO ][logstash.outputs.googlecloudstorage][main] Uploading file to my-logs/test/logstash_gcs_cw534756.local_2020-02-26T15:00.part036.log.gz but the logs are not ending up in the bucket. Removing /test from the bucket field uploads the logs to the root of the bucket (without a prefix).

Silently dropping data is not good.