I need to fetch objects from "requester pays" enabled S3 bucket in order to do so I have added "request_payer" => "requester" parameter to additional_settings field. However logstash does not recognize this option and throws the error below.
I can fetch the objects by using aws cli with "--request-payer requester" successfully. So there is nothing wrong with the bucket or access key configurations.
Could you please let me know whether S3 input plugin supports requester pays option?
Hello,
I need to fetch objects from "requester pays" enabled S3 bucket in order to do so I have added "request_payer" => "requester" parameter to additional_settings field. However logstash does not recognize this option and throws the error below.
If I remove "request_payer" => "requester" option, logstash returns access denied error as expected since AWS explicitly denotes that request payer field should be added to request header when requester pays option is enabled on the bucket. Reference: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ObjectsinRequesterPaysBuckets.html
I can fetch the objects by using aws cli with "--request-payer requester" successfully. So there is nothing wrong with the bucket or access key configurations.
Could you please let me know whether S3 input plugin supports requester pays option?
Thanks
Logstash Version: 7.10.2 S3 Input plugin version: 3.5.0
Config:
input { s3 { "access_key_id" => "xx" "secret_access_key" => "xx" "bucket" => "xx" "sincedb_path" => "xx" "prefix" => "xx" "region" => "xx" "additional_settings" => { "request_payer" => "requester" } } }
Error: [ERROR] 2021-03-09 16:55:57.519 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<ArgumentError: invalid configuration option"}
[INFO ] 2021-03-09 16:55:57.520 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[ERROR] 2021-03-09 16:55:57.524 [Converge PipelineAction::Create] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
:request_payer'>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/configuration.rb:166:in
block in apply_options'", "org/jruby/RubyHash.java:1415:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/configuration.rb:161:in
apply_options'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/configuration.rb:149:inbuild!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:68:in
build_config'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:19:ininitialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:105:in
new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/resource.rb:169:inextract_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/resource.rb:15:in
initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:417:inget_s3object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:96:in
register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:inblock in register_plugins'", "org/jruby/RubyArray.java:1809:in
each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:inregister_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:386:in
start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:311:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in
run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/home/xxx"], :thread=>"#