Get S3 Access Denied, Maybe because of configuration

[hykych]

I'm pretty sure the AK/SK, role_arn and bucket name are correct. I don't know if there is somethin wrong on the configuration. the configuration look like this:

<match kubernetes.**>
      @type copy

      # for debug
      # <store>
      #  @type file
      #  path /fluentd/docker.log
      #  format json
      #  include_time_key true
      #  time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      # </store>
      <store>
        @type s3
        @log_level debug

        auto_create_bucket false
        check_bucket false
        check_apikey_on_start false
        check_object false

        aws_key_id xxx
        aws_sec_key xxx
        s3_bucket bucket
        s3_region xxx
        path logs

        <assume_role_credentials>
          role_arn   xxx
          role_session_name xxx
        </assume_role_credentials>

        format json

        # if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
        # need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
        <buffer kube_uid,time>
          @type file
          path /fluent/s3_buffer
          timekey 60 # 1 min partition
          timekey_wait 1m
          timekey_use_utc true # use utc
          chunk_limit_size 256m
        </buffer>
      </store>

</match>

Get the fluentd logs look like this:

failed to flush the buffer. retry_time=0 next_retry_seconds=2020-05-25 07:57:05 +0000 chunk="5a6744c1ce918d02269eaf6329cb4f63" error_class=Aws::S3::Errors::AccessDenied error="Access Denied"

The role policy looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "s3:Put*",
                "s3:ListBucket",
                "s3:Get*",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucket/*",
                "arn:aws:s3:::bucket"
            ]
        }
    ]
}

[michaelkoro]

having the same issue, only my iam role has S3 full access policy. also, I'm using the instance_profile_credentials. have you managed to get it to work ?

[jotes]

Hey everyone! I've (probably) found the source of this problem. My instance of Fluentd has to use an IAM account and assume a role, similarly to @hykych's setup.

I think the problem lies in the function that authenticates Fluentd against a S3 bucket. When you specify IAM credentials, it skips the part about STS and doesn't assume a role. In my case, AWS S3 was denying access to my bucket.

I've managed to write a patch that fixes the issue for me: https://github.com/jotes/fluent-plugin-s3/commit/7f6d028a65edfb717830fd40b9dcd49c0604d0f3

I'm going to polish it and submit a PR.