aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.17k stars 313 forks source link

I want to be able to send different type of logs to different log groups #335

Open Unbreakable01 opened 5 years ago

Unbreakable01 commented 5 years ago

I want to send three different type of application logs to different log groups using AWS Log Driver to Cloud Watch Logs.

How do I configure that in task.json file. Suppose I have

  1. System Logs

  2. Audit Logs

  3. Trace Logs

and I want all of these to be in their own Log Groups. How do I configure that? Below is my Task.json file

      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "lt-stg-secops",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "transactions-stg-secops"
        }
yhlee-aws commented 5 years ago

Log drivers are configured at per-container level, so currently there isn't a way to branch out into multiple log groups. Marking this as feature request.

lbadger commented 5 years ago

I have been trying to figure out a way to do this. I currently support a legacy app that write's to many log files. It would be nice to pipe that all to stdout with a prefix and match on the prefix and write to multiple streams. I am currently using a sidecar container that is mounted to the logs folder. The log agent ends up shipping the logs to each of the streams. It works, but it's kind of a hacky solution.

PettitWesley commented 4 years ago

@Unbreakable01 Try out this tutorial; your use case might be solvable with FireLens (depends on the details of what your logs look like):

https://aws.amazon.com/blogs/opensource/splitting-application-logs-multiple-streams-fluent/

PettitWesley commented 3 years ago

In addition to that tutorial, this is also possibly relevant for some folks who might come to this issue: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/ecs-log-collection