logstash-plugins / logstash-output-s3

Apache License 2.0
58 stars 151 forks source link

upload_queue_size doesnt do anything #225

Open marielcherkassky opened 4 years ago

marielcherkassky commented 4 years ago

Hey, I was trying to upload to s3 each message that I get from my input without doing any aggregation. Maybe I got it wrong but according to the docs upload_queue_size sounds fit for this issue - Number of items we can keep in the local queue before uploading them.

The output pipeline I configured :

output {
  s3 {
    region => "eu-east-1"
    bucket => "my_bucket"
    upload_queue_size => 1
    prefix => "files/"
    temporary_directory => "/etc/logstash/tmp_files_to_s3"
  }
  file {
     codec => rubydebug
     path => "/tmp/monitor.log"
  }
}

By default the rotation_strategy is configured to time or size thresholds. So I dont see how it behaves together with the upload_queue_size when it is configured.