search

fluent / fluent-plugin-s3

Amazon S3 input and output plugin for Fluentd
https://docs.fluentd.org/output/s3
314 stars 217 forks source link

fluent-plugin-s3 doesn't seem to work with AWS short-term credentials #427

Open sanjay-curium opened 1 year ago

sanjay-curium commented 1 year ago

Describe the bug

I have been trying to upload aggregated logs through fluentD to an s3 bucket. The entire set-up works if I use AWS long-term access keys but not with the short-term credentials. The error thrown looks like the following.

unexpected error error_class=RuntimeError error="can't call S3 API. Please check your credentials or s3_region configuration. error = #<Aws::S3::Errors::InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.>"

To Reproduce

  1. Build a docker image for the fluentD using the following contents of the dockerfile. 
    FROM fluentd:latest
USER root
RUN fluent-gem install fluent-plugin-s3
COPY fluentD.conf /fluentd/etc/fluent.conf
  2. My fluentD.conflooks like this. 
    
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
@type http
port 9880
bind 0.0.0.0

<match .> @type s3 aws_key_id "#{ENV['AWS_ACCESS_KEY_ID']}" aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}" aws_sso_key "#{ENV['AWS_SSO_KEY']}" s3_bucket "#{ENV['BUCKET_NAME']}" s3_region "#{ENV['AWS_REGION']}"

<buffer tag,time> @type file path /output/test.log timekey 60 # 1 minute partition timekey_wait 10s timekey_use_utc true chunk_limit_size 256m append true 

3. Run the container using the following command.

docker run -p 9880:9880 -p 24224:24224 -v ./logs:/output/ -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_SSO_KEY=$AWS_SESSION_TOKEN -e BUCKET_NAME=$BUCKET_NAME -e AWS_REGION=$AWS_REGION image_name:tag


### Expected behavior

Logs show up on the s3 bucket if I use AWS long-term credentials but not with the short-term ones.

### Your Environment

```markdown
- Fluentd version: gem 'fluentd' version '1.16.0' -- `fluentd:latest` docker image.
- s3 plugin version: gem 'fluent-plugin-s3' version '1.7.2'

Your Configuration

@type forward port 24224 bind 0.0.0.0 

@type http
port 9880
bind 0.0.0.0

<match .> @type s3 aws_key_id "#{ENV['AWS_ACCESS_KEY_ID']}" aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}" aws_sso_key "#{ENV['AWS_SSO_KEY']}" s3_bucket "#{ENV['BUCKET_NAME']}" s3_region "#{ENV['AWS_REGION']}"

<buffer tag,time> @type file path /output/test.log timekey 60 # 1 minute partition timekey_wait 10s timekey_use_utc true chunk_limit_size 256m append true

Your Error Log

2023-08-02 04:28:06 +0000 [error]: #0 unexpected error error_class=RuntimeError error="can't call S3 API. Please check your credentials or s3_region configuration. error = #<Aws::S3::Errors::InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.>"

Additional context

No response