fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.73k stars 1.56k forks source link

Add FIPS endpoint support for AWS S3 Output Plugin #9098

Open jerisalan opened 1 month ago

jerisalan commented 1 month ago

Is your feature request related to a problem? Please describe. We heavily use the fluent-bit plugin to process and then route some of our container logs to S3 buckets for storage and further analysis. It works great for many scenarios and but for compliance needs, we would like to have the ability to have the plugin send logs via the S3 FIPS endpoints for supported AWS regions.

We have tried using the endpoint property (to something like https://$S3BUCKET.s3-fips.$REGION.amazonaws.com) that comes with S3 Configuration Parameters but to no avail. The PutObject AWS API call still resolves to a non-FIPS endpoint.

https://docs.aws.amazon.com/general/latest/gr/s3.html https://aws.amazon.com/compliance/fips/

Describe the solution you'd like The S3 output plugin intended for uploading objects to AWS S3 allows for FIPS endpoints to be set as a configuration parameter so that PutObject requests get serviced via AWS S3 FIPS endpoints.

Describe alternatives you've considered None

Additional context

[OUTPUT]
   Name               s3
   Match              td.*
   bucket             $S3_BUCKET
   region             $AWS_REGION
   fips                 true <- introduce a new property like this
   total_file_size    100M
   upload_timeout     1m
   use_put_object     On
   static_file_path   false
   s3_key_format_tag_delimiters .
   content_type       text/plain
   json_date_key      false
   store_dir          /var/fluent-bit
   s3_key_format      /json/$TAG[1]/$TAG[2]/$TAG[3]-%H%M%S-$UUID.json
Athishpranav2003 commented 1 month ago

@jerisalan Seems like the S3 plugin already has support to add custom endpoint. So i guess this change can be achieved by simply specifying the endpoint explicitly. https://docs.fluentbit.io/manual/pipeline/outputs/s3#configuration-parameters . Can you please check if this solves the purpose