logstash-plugins / logstash-output-google_cloud_storage

Apache License 2.0
9 stars 25 forks source link

log_file_prefix and date_pattern don't support the use of forward slashes #47

Open JoshMock opened 3 years ago

JoshMock commented 3 years ago

Logstash information:

Please include the following information:

  1. Logstash version: 7.13.2
  2. Logstash installation source: Docker
  3. How is Logstash being run: Kubernetes
  4. How was the Logstash Plugin installed: Dockerfile line: RUN ./bin/logstash-plugin install logstash-output-google_cloud_storage

OS version Linux kernel 4.14.174 (I believe the Docker image is based on CentOS)

Description of the problem including expected versus actual behavior:

Putting a / character in log_file_prefix (e.g. log_file_prefix => "my-prefix/") causes i/o to the temp_directory location to fail with errors like the following:

[2021-08-20T10:00:00.000][ERROR][logstash.javapipeline    ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Errno::ENOENT: No such file or directory - /tmp/logstash-gcs/my-prefix/logstash_gcs_cw054563.local_2021-08-20T10:00.part001.log.gz>, :backtrace=>["org/jruby/RubyIO.java:1239:in `sysopen'"

Presumably this is because it's trying to write a file into a subdirectory of temp_directory that doesn't exist yet, assuming that all temp files should be written to one flat directory, and uploaded to GCS as such as well. Similar symptoms arise when putting forward slashes in date_pattern.

See https://github.com/logstash-plugins/logstash-output-google_cloud_storage/issues/45 and https://github.com/logstash-plugins/logstash-output-google_cloud_storage/issues/46 for further examples of this.

Steps to reproduce:

A pipeline to reproduce:

input { generator { } }
output {
  google_cloud_storage {
    bucket => "my-bucket"
    json_key_file => "/secrets/credentials.json"
    log_file_prefix => "my-prefix/"
  }
}
pspot2 commented 10 months ago

Analogous parameter implementation in S3 plugin for reference: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html#plugins-outputs-s3-prefix