search

logstash-plugins / logstash-input-s3

Apache License 2.0
57 stars 150 forks source link

Could not fetch objects from "requester pays" enabled bucket #222

Open dkayahan opened 3 years ago

dkayahan commented 3 years ago

Hello,

I need to fetch objects from "requester pays" enabled S3 bucket in order to do so I have added "request_payer" => "requester" parameter to additional_settings field. However logstash does not recognize this option and throws the error below.

If I remove "request_payer" => "requester" option, logstash returns access denied error as expected since AWS explicitly denotes that request payer field should be added to request header when requester pays option is enabled on the bucket. Reference: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ObjectsinRequesterPaysBuckets.html

I can fetch the objects by using aws cli with "--request-payer requester" successfully. So there is nothing wrong with the bucket or access key configurations.

Could you please let me know whether S3 input plugin supports requester pays option?

Thanks

Logstash Version: 7.10.2 S3 Input plugin version: 3.5.0

Config:

input { s3 { "access_key_id" => "xx" "secret_access_key" => "xx" "bucket" => "xx" "sincedb_path" => "xx" "prefix" => "xx" "region" => "xx" "additional_settings" => { "request_payer" => "requester" } } }

Error: [ERROR] 2021-03-09 16:55:57.519 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<ArgumentError: invalid configuration option :request_payer'>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/configuration.rb:166:inblock in apply_options'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/configuration.rb:161:inapply_options'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/configuration.rb:149:in build!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:68:inbuild_config'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:19:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:105:innew'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/resource.rb:169:in extract_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/resource.rb:15:ininitialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:417:in get_s3object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:96:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in block in register_plugins'", "org/jruby/RubyArray.java:1809:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:386:instart_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:311:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/home/xxx"], :thread=>"#"} [INFO ] 2021-03-09 16:55:57.520 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"} [ERROR] 2021-03-09 16:55:57.524 [Converge PipelineAction::Create

] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create
, action_result: false", :backtrace=>nil}

dkayahan commented 3 years ago

I have resolved it by manually adding required header field to aws-sdk-core signer api but a high level solution would be appreciated if there is any.