search

aws / amazon-cloudwatch-logs-for-fluent-bit

A Fluent Bit output plugin for CloudWatch Logs
Apache License 2.0
174 stars 49 forks source link

[Feature Request] Allow to create log-groups with kms encryption #119

Open markussiebert opened 4 years ago

markussiebert commented 4 years ago

At the moment I can't find an option for creating kms encrypted log groups, but I think it would be an excellent feature.

It should be possible via golang to create an encrypted log group: https://docs.aws.amazon.com/sdk-for-go/api/service/cloudwatchlogs/#CreateLogGroupInput

PettitWesley commented 1 year ago

Typically a best practice is to create log groups with infrastructure as code (CloudFormation or TerraForm or CDK or etc). This makes it easy to set centralized policies for your teams on retention, and also encryption. It also makes it easy to delete log groups. Whereas, if you have Fluent Bit create them, there is nothing tracking all of the log groups it creates.

However, we do understand that some folks want to create the log groups with Fluent Bit, and so we have added this request to our backlog. I am uncertain as to when we would pick it up though.

This request is an ideal one for a new/novice contributor who wants to add an option. You can see the past commit where log retention setting was added, this feature is similar.

etoews commented 1 year ago

FWIW ... for our use case, we're using this via an addon in terraform-aws-eks-blueprints. We need the LogGroups that ultimately get created by it dynamically to be encrypted with KMS.

PettitWesley commented 1 year ago

@etoews I see this in their docs (which is just a wrapper around the AWS for Fluent Bit helm chart from what I can tell https://github.com/aws/eks-charts/tree/master/stable/aws-for-fluent-bit): https://aws-ia.github.io/terraform-aws-eks-blueprints/v4.14.0/add-ons/aws-for-fluent-bit/#externally-created-cloudwatch-log-groups

Why does that not allow you to accomplish what you want?

etoews commented 1 year ago

@PettitWesley That's for externally created LogGroups. Like the OP, we are not externally creating LogGroups. FluentBit is creating the LogGroups for us and hence the need to allow to create LogGroups with KMS encryption.

PettitWesley commented 1 year ago

@etoews I understand the request. And we have this on our long term backlog. I understand that your existing workflow is to have Fluent Bit create the log groups, and that you would like to enable KMS encryption without modifying your existing workflow.

However, I want to note that a workaround exists that is compatible with the helm chart. And IMO, creating the log group externally via infra as code is generally the best practice.

Let me know if I have incorrectly summarized the status of this use case.

etoews commented 1 year ago

@PettitWesley That correctly summarises it.