search

cherweg / logstash-input-s3-sns-sqs

logstash input downloading files from s3 Bucket by OjectKey from SNS/SQS
Other
29 stars 35 forks source link

Using a pipeline without explicit aws credentials fails... #19

Closed christianherweg0807 closed 5 years ago

christianherweg0807 commented 5 years ago

I have a pipeline without s3_role or iam_access_key... After starting logstash i got the following error:

[2019-06-20T16:43:09,416][WARN ][logstash.inputs.s3snssqs ] Error in poller loop {:error=>#<Aws::Errors::MissingCredentialsError: unable to sign request without credentials set>}
[2019-06-20T16:43:09,416][WARN ][logstash.inputs.s3snssqs ] Backtrace:
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/request_signer.rb:104:in `require_credentials'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_request_signer.rb:14:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_host_id.rb:14:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_request_signer.rb:65:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_redirects.rb:15:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/retry_errors.rb:108:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_dualstack.rb:32:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_accelerate.rb:49:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_md5s.rb:31:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/rest/handler.rb:7:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/endpoint_pattern.rb:27:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/endpoint_discovery.rb:67:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/seahorse/client/plugins/endpoint.rb:41:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/seahorse/client/plugins/response_target.rb:21:in `call'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/seahorse/client/request.rb:70:in `send_request'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/seahorse/client/base.rb:207:in `block in define_operation_methods'
    /Users/cherweg/git/forks/logstash-input-s3-sns-sqs/lib/logstash/inputs/s3/downloader.rb:19:in `block in copy_s3object_to_disk'
    /Users/cherweg/git/forks/logstash-input-s3-sns-sqs/lib/logstash/inputs/s3/client_factory.rb:40:in `get_s3_client'
    /Users/cherweg/git/forks/logstash-input-s3-sns-sqs/lib/logstash/inputs/s3/downloader.rb:18:in `copy_s3object_to_disk'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/local_gems/bb10b1f0/logstash-input-s3-sns-sqs-2.0.0/lib/logstash/inputs/s3snssqs.rb:302:in `block in run_worker_thread'
    /Users/cherweg/git/forks/logstash-input-s3-sns-sqs/lib/logstash/inputs/sqs/poller.rb:94:in `block in run'
    /Users/cherweg/git/forks/logstash-input-s3-sns-sqs/lib/logstash/inputs/sqs/poller.rb:134:in `block in preprocess'
    org/jruby/RubyArray.java:1734:in `each'
    /Users/cherweg/git/forks/logstash-input-s3-sns-sqs/lib/logstash/inputs/sqs/poller.rb:121:in `preprocess'
    /Users/cherweg/git/forks/logstash-input-s3-sns-sqs/lib/logstash/inputs/sqs/poller.rb:92:in `block in run'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.177/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413:in `block in yield_messages'
    org/jruby/RubyArray.java:1734:in `each'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.177/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412:in `yield_messages'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.177/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405:in `block in process_messages'
    org/jruby/RubyKernel.java:1114:in `catch'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.177/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404:in `process_messages'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.177/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336:in `block in poll'
    org/jruby/RubyKernel.java:1292:in `loop'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.177/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331:in `block in poll'
    org/jruby/RubyKernel.java:1114:in `catch'
    /Users/cherweg/git/logstash/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.11.177/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330:in `poll'
    /Users/cherweg/git/forks/logstash-input-s3-sns-sqs/lib/logstash/inputs/sqs/poller.rb:82:in `block in run'
    /Users/cherweg/git/forks/logstash-input-s3-sns-sqs/lib/logstash/inputs/sqs/poller.rb:163:in `run_with_backoff'
    /Users/cherweg/git/forks/logstash-input-s3-sns-sqs/lib/logstash/inputs/sqs/poller.rb:81:in `run'