cherweg / logstash-input-s3-sns-sqs

logstash input downloading files from s3 Bucket by OjectKey from SNS/SQS
Other
29 stars 35 forks source link

Couple of Issues #13

Closed saocean closed 5 years ago

saocean commented 5 years ago

I get an error when running rspec (below). No issues with bundle install. Definitely a n00b at jruby. I was able to still make the GEM file and install the plugin in Logstash, but I just want to cover all my bases.

Also, two questions:

  1. I wanted to see if there was anything to stop this plugin from being used cross account using assume role. I have an IAM policy set up, pretty much duplicating your example IAM policy but also attaching a trust policy. I have been able to manually get sqs messages using the AWS cli with no issues. Just wanted to make sure this was a supported config.

  2. Also, would there be any issues using AWS GovCloud region for example us-gov-west-1 and and an SQS queue that resides in that region, for example: arn:aws-us-gov:sqs:us-gov-west-1:accountid:queuename

I appreciate any help you can provide.

Thanks,

Here is my command output: jruby -S bundle exec rspec

An error occurred while loading ./spec/inputs/s3sqs_spec.rb. Failure/Error: # encoding: utf-8

RuntimeError:

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

no such file to load -- com/fasterxml/jackson/module/jackson-module-afterburner/2.7.4/jackson-module-afterburner-2.7.4 (LoadError)

./spec/inputs/s3sqs_spec.rb:1:in `(root)'

./spec/inputs/s3sqs_spec.rb:2:in `
'

------------------

--- Caused by: ---

LoadError:

no such file to load -- com/fasterxml/jackson/module/jackson-module-afterburner/2.7.4/jackson-module-afterburner-2.7.4

./spec/inputs/s3sqs_spec.rb:1:in `(root)'

No examples found.

Finished in 0.0043 seconds (files took 0.63726 seconds to load)

Also

cherweg commented 5 years ago

I‘m on holiday until next week.

Regards Christian

Von meinem iPhone gesendet

Am 26.03.2019 um 20:10 schrieb MikE notifications@github.com:

I get an error when running rspec (below). No issues with bundle install. Definitely a n00b at jruby. I was able to still make the GEM file and install the plugin in Logstash, but I just want to cover all my bases.

Also, two questions:

I wanted to see if there was anything to stop this plugin from being used cross account using assume role. I have an IAM policy set up, pretty much duplicating your example IAM policy but also attaching a trust policy. I have been able to manually get sqs messages using the AWS cli with no issues. Just wanted to make sure this was a supported config.

Also, would there be any issues using AWS GovCloud region for example us-gov-west-1 and and an SQS queue that resides in that region, for example: arn:aws-us-gov:sqs:us-gov-west-1:accountid:queuename

I appreciate any help you can provide.

Thanks,

Here is my command output: jruby -S bundle exec rspec

An error occurred while loading ./spec/inputs/s3sqs_spec.rb. Failure/Error: # encoding: utf-8

RuntimeError:

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command no such file to load -- com/fasterxml/jackson/module/jackson-module-afterburner/2.7.4/jackson-module-afterburner-2.7.4 (LoadError)

./spec/inputs/s3sqs_spec.rb:1:in `(root)'

./spec/inputs/s3sqs_spec.rb:2:in `'


--- Caused by: ---

LoadError:

no such file to load -- com/fasterxml/jackson/module/jackson-module-afterburner/2.7.4/jackson-module-afterburner-2.7.4

./spec/inputs/s3sqs_spec.rb:1:in `(root)'

No examples found.

Finished in 0.0043 seconds (files took 0.63726 seconds to load)

Also

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

saocean commented 5 years ago

No worries, whenever you get a chance.

Also the reason i mention the GovCloud item is because i think the plugin is defaulting to regular AWS and not using the region as part of the the URL for the SQS queue.

I receive the below error, but using the aws cli on the same logstash machine, i can pull data from the queue with no issues.

][ERROR][logstash.inputs.s3snssqs ] Cannot establish connection to Amazon SQS {:error=>#}

dhammond22222 commented 5 years ago

Any updates on the assume role functionality for cross account access?

saocean commented 5 years ago

We were able to resolve. I had to specify our role assumption twice using the s3_role_arn as well as the native logstash-input-sqs parameter "role_arn" which I'm assuming is what sqs used separately to assume the role. Hope this helps everyone out.