search

aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.21k stars 319 forks source link

[ECS] [request]: AWS firelens config missing options for S3 and path #946

Open Oegma2 opened 4 years ago

Oegma2 commented 4 years ago

Community Note

Tell us about your request On the task-definition page, for firelens configuration for fluentbit, we need additional fields for advanced routing options - currently, the option to change the config to file or S3 isn't available from the AWS console. Some of the fluentbit routing magic lies in the ability to use custom config files with different outputs, but in order to use those features, we need to change the options to file. I've attached two screenshots below to provide some guidance where a good place would be for those new fields and which config values I am referring to

Below is a suggestion for where we can place those extra required fields aws_firelens_options

Where in the task-definition file are those fields then? - see screenshot below aws_firelens_options2

Which service(s) is this request for? This could be Fargate, ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Trying to setup fargate containers with firelens using fluentbit as log-router, shipping logs to multi-outputs. In our case, cloudwatch and datadog. In order to route logs to multiple outputs, require a custom config file for fluentbit, and at this stage, the AWS console for creating the task-definition doesn't provide those features to specify S3 or File as config.

Are you currently working around this issue? No solution at this stage - one option we persuing is to create a custom config file, upload that as part of a custom fluentbit image and use the aws cli to create a task-definition with the options set to file

Additional context Also struggled with using the aws cli task-definition setting the fluentbit content to S3 - meaning pull the custom config file from S3 ... the fargate container crash all the time when using a config file hosted on S3 - even after adding the ECS role full access to S3, still failed. That same config file work perfectly fine in a normal fluentbit startup, but placed on S3, and using the task-definition with firelens options set to use S3 never worked. - our last resort will be changing this to file

First created a request on this GitHub page but closed it - makes more sense for the container roadmap - https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/issues/52

khangly commented 2 years ago

In FirelensConfiguration, it says you can't use S3 for the config file if you use Fargate.

PettitWesley commented 2 years ago

This newly released feature inside of AWS for Fluent Bit serves the same use case and provides additional features: https://github.com/aws/aws-for-fluent-bit/blob/mainline/use_cases/init-process-for-fluent-bit/README.md

You can pull multiple config files from S3 and also get ECS Metadata via env vars.

To be clear, the S3 config-file-type is still not supported on Fargate. Instead of building that, we decided to build an init script (implemented in Golang) into AWS for Fluent Bit (currently in a different image tag than the existing tags). In retrospect, I think config sourcing is something that is better and easier to do inside the log router itself.

Please try it and let us know what you think.

https://github.com/aws/aws-for-fluent-bit#using-the-init-tag