search

logstash-plugins / logstash-input-s3

Apache License 2.0
57 stars 150 forks source link

S3 input plugin is not reading AWS-KMS (CMK) encrypted bucket #216

Open deepanshumarwah opened 3 years ago

deepanshumarwah commented 3 years ago

I am running a logstash container with s3 pipeline as per below configuration:

input { s3 { id => "pipeline_s3_example_bucket_input" bucket => "example-bucket" region => "ap-southeast-1" access_key_id => "#######################" secret_access_key => "#######################" codec => "json_lines" sincedb_path => "/sincedbs/pipeline_s3_example_bucket.sincedb" prefix => "folderA" add_field => { "type" => "example-bucket-logs" "host" => "example-bucket" } } }

My s3 bucket is encrypted with AWS-KMS using a custom managed key. I am using below IAM policy for user to read the bucket data :

"Version": "2012-10-17", "Statement": [ { "Sid": "IAMPolicy", "Effect": "Allow", "Action": [ "s3:GetObject", "kms:Decrypt", "s3:GetBucketLogging", "kms:GenerateDataKey", "kms:DescribeKey", "s3:GetObjectTagging", "s3:ListBucket", "s3:GetBucketVersioning", "s3:GetBucketLocation", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:kms:ap-southeast-1:<account-id>:key/<Key-ID>", "arn:aws:s3:::example-bucket", "arn:aws:s3:::example-bucket/*" ] } ] } I also added the user arn in the Key users, I tested the access using aws cli commands and that worked as well. However, logstash is not able to pull those logs and I cant see any error in container logs as well. Please help if anyone else has faced similar issue.

Link to elastic community : https://discuss.elastic.co/t/s3-input-plugin-is-not-reading-aws-kms-cmk-encrypted-bucket/249439

kaisecheng commented 3 years ago

I marked it as an enhancement to support kms encrypted file It requires extra headers

GET /example_image.jpg HTTP/1.1
Host: example-bucket.s3.amazonaws.com  
Accept: */*  
Authorization: 5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7
Date: Wed, 28 May 2014 19:31:11 +0000  
x-amz-server-side-encryption: aws:kms 
x-amz-server-side-encryption-aws-kms-key-id:arn:aws:kms:us-east-1:111122223333:key/0695f802-503c-40n2-d17d-16d702f79f01
Z4ck404 commented 11 months ago

Are there any updates about this issue ?