Open deepanshumarwah opened 3 years ago
I marked it as an enhancement to support kms encrypted file It requires extra headers
GET /example_image.jpg HTTP/1.1
Host: example-bucket.s3.amazonaws.com
Accept: */*
Authorization: 5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7
Date: Wed, 28 May 2014 19:31:11 +0000
x-amz-server-side-encryption: aws:kms
x-amz-server-side-encryption-aws-kms-key-id:arn:aws:kms:us-east-1:111122223333:key/0695f802-503c-40n2-d17d-16d702f79f01
Are there any updates about this issue ?
I am running a logstash container with s3 pipeline as per below configuration:
input {
s3 {
id => "pipeline_s3_example_bucket_input"
bucket => "example-bucket"
region => "ap-southeast-1"
access_key_id => "#######################"
secret_access_key => "#######################"
codec => "json_lines"
sincedb_path => "/sincedbs/pipeline_s3_example_bucket.sincedb"
prefix => "folderA"
add_field => {
"type" => "example-bucket-logs"
"host" => "example-bucket"
}
}
}
My s3 bucket is encrypted with AWS-KMS using a custom managed key. I am using below IAM policy for user to read the bucket data :
"Version": "2012-10-17",
"Statement": [
{
"Sid": "IAMPolicy",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"kms:Decrypt",
"s3:GetBucketLogging",
"kms:GenerateDataKey",
"kms:DescribeKey",
"s3:GetObjectTagging",
"s3:ListBucket",
"s3:GetBucketVersioning",
"s3:GetBucketLocation",
"s3:GetObjectVersion"
],
"Resource": [
"arn:aws:kms:ap-southeast-1:<account-id>:key/<Key-ID>",
"arn:aws:s3:::example-bucket",
"arn:aws:s3:::example-bucket/*"
]
}
]
}
I also added the user arn in the Key users, I tested the access using aws cli commands and that worked as well. However, logstash is not able to pull those logs and I cant see any error in container logs as well. Please help if anyone else has faced similar issue.Link to elastic community : https://discuss.elastic.co/t/s3-input-plugin-is-not-reading-aws-kms-cmk-encrypted-bucket/249439