search

fluent / fluent-plugin-s3

Amazon S3 input and output plugin for Fluentd
https://docs.fluentd.org/output/s3
314 stars 218 forks source link

match PR122 for input, finish cross-account bucket/sqs capability with assumeRole #249

Closed jordo1138 closed 6 years ago

jordo1138 commented 6 years ago

Signed-off-by: Jordan Hamel jordanhamel@gmail.com

jordo1138 commented 6 years ago

122 only fixed for output, this should work also for input

jordo1138 commented 6 years ago

@cosmo0920 hi, I tested this locally and it's giving me the right region now without any error, but I seem to have come across a different issue that seems specific to using assumeRole, the sqs queue name is nil so the sqs post goes to the right region but is / instead of the sqs queue name... this should be because the queue name seems to append after the account number of the aws account, but in this case the queue may not exist in the aws account of the assumed role if they are granted access to a cross account queue..either way it seems related to assumeRole behavior

example: if using aws key and secret key

POST /12234242424/queue_name

if using assume role credentials:

POST /

fluentd -vv shows this, and has the correct queue_name from the fluent.conf, bug of course is 400 error as POST is to /

2018-11-06 07:55:00 +0000 [info]: #0 fluent/log.rb:322:info: [Aws::SQS::Client 400 0.255928 0 retries] get_queue_url(queue_name:"xxxxxxxx") Aws::SQS::Errors::NonExistentQueue The specified queue does not exist for this wsdl version.

I opened a new issue #250 with more details on this separate problem

jordo1138 commented 6 years ago

@cosmo0920 @repeatedly hi, so i have added the changes to the input plugin to allow for cross account ID of owner of the sqs to be passed. Based on the ruby sdk docs here https://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#get_queue_url-instance_method I modified the tests also to look for the dummy account id 123456789123 please let me know if you have any feedback.. I left it under this PR since its all related actually to allowing cross account input work

repeatedly commented 6 years ago

Merged. Thanks!