search

aws / aws-for-fluent-bit

The source of the amazon/aws-for-fluent-bit container image
Apache License 2.0
455 stars 135 forks source link

Custom Fluent Bit in AWS EKS on Fargate #728

Open chinmaychandak opened 1 year ago

chinmaychandak commented 1 year ago

Describe the question/issue

I want to customize the Fluent Bit image used in the EKS on Fargate setup.

I set up logging to S3 via Kinesis Data Firehose as documented here: https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html - i.e. using an aws-logging ConfigMap in the aws-observability namespace.

But I now want to modify the Fluent Bit config to add filesystem-based buffering, a Lua filter, and maybe even write to a custom HTTP endpoint.

Doing a bit of research, I found that ECS on Fargate can do all of the above using a custom FLB image:

But I could not find how to specify a custom FLB image in EKS on Fargate.

Is this possible at all? Can someone please help me out?

DrewZhang13 commented 1 year ago

Yeah we could achieve that, previously you had to run a sidecar to route container logs from Amazon EKS pods running on AWS Fargate, you can now use a built-in log router. check 2 blogs for details

chinmaychandak commented 1 year ago

Thanks @DrewZhang13, really appreciate the help!

I did go through these blogs but there does not seem to be a way to specify a custom FLB image. There is not a lot of customizations available because AWS-managed FLB used in the built-in log router does not allow things like Lua Filters, custom Input and Output plugins, etc.

Did I miss anything?

PettitWesley commented 1 year ago

@chinmaychandak Yes there is no way to customize the image for the built-in log router, and as you noted, there's very limited support for only specific plugins.

If you want full control, then follow this blog to deploy Fluent Bit as a sidecar you manage: https://aws.amazon.com/blogs/containers/how-to-capture-application-logs-when-using-amazon-eks-on-aws-fargate/

chinmaychandak commented 1 year ago

Thank you so much for the help, @PettitWesley! Is there a roadmap for allowing such customizations in the built-in log router?

PettitWesley commented 1 year ago

@chinmaychandak I do not think so, but you can check: https://github.com/aws/containers-roadmap