cherweg / logstash-input-s3-sns-sqs

logstash input downloading files from s3 Bucket by OjectKey from SNS/SQS
Other
29 stars 35 forks source link

role_arn is not working #50

Open JayakumarRamesh opened 4 years ago

JayakumarRamesh commented 4 years ago

input { s3snssqs { region => "us-east-1" queue => "" queue_owner_aws_account_id => "" access_key_id => "*****" secret_access_key => "**" role_arn => "arn:aws:iam::accountid:role/rolename" role_session_name => "logstash"

sqs_skip_delete => true

codec                      => line
from_sns                   => false
temporary_directory        => "Temp"
s3_access_key_id              => "**************"
s3_secret_access_key          => "*************" 
s3_role_arn                   => "arn:aws:iam::accountid:role/rolename"
s3_options_by_bucket => [
                            {
                                bucket_name => "*************"
                            }
                        ]

}
}

output { stdout { } }

I am using role_arn to fetch data from another aws account but I am getting error though my credentials and assumed role are correct.

Error:

[2020-09-10T21:45:50,635][ERROR][logstash.inputs.s3snssqs ][main] Cannot establish connection to Amazon SQS {:error=># you do not have access to it.>} [2020-09-10T21:45:50,644][DEBUG][logstash.javapipeline ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#"} [2020-09-10T21:45:50,649][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"} [2020-09-10T21:45:50,651][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"} [2020-09-10T21:45:50,731][DEBUG][logstash.javapipeline ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#"} [2020-09-10T21:45:50,733][DEBUG][logstash.javapipeline ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#"} [2020-09-10T21:45:50,734][DEBUG][logstash.javapipeline ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#"} [2020-09-10T21:45:50,741][DEBUG][logstash.outputs.stdout ][main] Closing {:plugin=>"LogStash::Outputs::Stdout"} [2020-09-10T21:45:50,748][DEBUG][logstash.pluginmetadata ][main] Removing metadata for plugin 6e3e4da5af4891d1493e09316ac7bbe9afccc27748b9230d02cc89b3eebdce81 [2020-09-10T21:45:50,752][ERROR][logstash.javapipeline ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=># your credentials>, :backtrace=>["D:/logstash/sqslog/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:58:in initi alize'", "D:/logstash/sqslog/logstash-7.8.0/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:251:inregister'", "D:/logstash/sqslo g/logstash-7.8.0/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:216:in block in register_plugins'", "org/jruby/RubyArray.java:1809:ineach'", "D:/logstash/sqslog/logstash-7.8.0/l ogstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:215:in register_plugins'", "D:/logstash/sqslog/logstash-7.8.0/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:326:instar t_inputs'", "D:/logstash/sqslog/logstash-7.8.0/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:286:in start_workers'", "D:/logstash/sqslog/logstash-7.8.0/logstash-7.8.0/logstash-co re/lib/logstash/java_pipeline.rb:170:inrun'", "D:/logstash/sqslog/logstash-7.8.0/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:125:in `block in start'"], "pipeline.sources"=>["D :/logstash/sqslog/logstash-7.8.0/logstash-7.8.0/pipelines/s3snssqs.conf"], :thread=>"#"} [2020-09-10T21:45:50,777][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: Pip elineAction::Create, action_result: false", :backtrace=>nil}

Please help me to resolve this issue.

christianherweg0807 commented 4 years ago

s3_role_arn is removed with 2.1 i think... Please use the example in the docs: https://github.com/cherweg/logstash-input-s3-sns-sqs/blob/master/docs/index.asciidoc

    s3_options_by_bucket       => [
        { bucket_name => "logs-bucket-222222222222-.*"
          credentials => { role => "arn:aws:iam::222222222222:role/logging-role" }
        },
hinchliff commented 3 years ago

The docs suggest that you only need to specify the credentials if you are overwriting "the default" ?

If you have credentials per s3 bucket you could overwrite the default.

But it doesn't look like you can put credentials in the top-level settings?

[2021-09-22T18:58:00,742][ERROR][logstash.inputs.s3snssqs ] Unknown setting 'credentials' for s3snssqs

So you need to specify them separately for each bucket, even if they all use the same setting?