fluent / fluent-plugin-s3

Amazon S3 input and output plugin for Fluentd
https://docs.fluentd.org/output/s3
314 stars 218 forks source link

if aws_key_id and aws_sec_key are declared, no assume role takes place #247

Closed jordo1138 closed 5 years ago

jordo1138 commented 6 years ago

I have found that for s3 input, if you declare the aws_key_id and aws_sec_key, they do not then get used in order to try and assume the role given. In my my case, my readonly role for a cross-account bucket was granted to my iam user which has the key and secret key I put. But seems the s3 api tries with the directly with the key and secret key and doesn't not try to assume the role given. In the docs, it does mention that the key should be provided if using on ec2 without iam role, which is true in my case as the ec2 running fluentd has no IAM role attached, but cannot handle the case where my iam user is provided and should also then assume the cross account role that can read the cross account bucket... """ config:

<source>
    @type s3
    aws_key_id xxxx  ##in account A
    aws_sec_key xxxx
    s3_bucket "some-bucket"  ## in account B, aws_key_id cannot access the bucket
    s3_region "us-west-2"
    tag "cloudtrail.all"
    <assume_role_credentials>
      role_arn "arn:aws:iam::xxxxxxxx:role/xxxxReadOnly"  ## role in account B can access bucket
      role_session_name "fluentd"
    </assume_role_credentials>
    <sqs>
      queue_name "xxxxxcldtrail"
    </sqs>
</source>

fails with access denied (not assuming the role at all) cloudtrail log shows the denial comes to the aws_key_id account, which is expected as it doesn't have rights to the s3 bucket, only the cross-account role has it

is this a feature request then? bug? If I had to use assume role, the docs seem to say that i need to have instance profile creds, which implies iam role attached to the instance. Correct me if I"m wrong @repeatedly thanks so much for your help so far. It's working great with my test sqs and bucket for my aws account where assumeRole isn't needed. Right now issue is this since I have to reach a bucket with many accounts cloudtrail logs

jordo1138 commented 6 years ago

Perhaps the obvious workaround is to grant assumeRole rights to a new iam role in account a and attach to the ec2 instance?

jordo1138 commented 6 years ago

separate issue, when I did go to use assumeRole for input I"m getting the same "region not set" error that seems to have been fixed with #122 back in 2015 but was only fixed for out, not input.rb @okkez

249 pull request opened for matching in/out behavior with assumeRole

jordo1138 commented 5 years ago

Closed after #249