search

aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.21k stars 317 forks source link

Enable support for custom log drivers #435

Open dangerousplay opened 5 years ago

dangerousplay commented 5 years ago

Summary

Hello, our company needs to use a custom log driver designed to meet our requirements, but unfortunately, the ECS agent does not support custom log drivers.

Description

Because Docker's built-in gelf log driver does not meet all of our company's requirements, we have developed a plugin for Docker, a custom log driver that transforms and handles the logs of applications running on Docker. One of the transformations is the aggregation of n partial logs into a single log containing the entire message. We also have resolved a known TTL DNS issue that affects the log driver gelf (Reported at issue https://github.com/moby/moby/issues/17904#issuecomment-194403940). Our solution is working, but when we tried to use the log driver on the ECS agent, we figure out that it does not support custom log drivers.

Maybe something should be done in this session code:

https://github.com/aws/amazon-ecs-agent/blob/f95f731bce74e6276f9fd8bfbb87a62d3a790ce8/agent/config/config.go#L295-L304

shubham2892 commented 5 years ago

Hi @dangerousplay, unfortunately we currently do not support custom log drivers.

I will mark this as a feature request.

hermes-pimentel commented 5 years ago

Hi @dangerousplay, unfortunately we currently do not support custom log drivers.

I will mark this as a feature request.

Sumoligic is a custom log driver and is supported...

petderek commented 5 years ago

I think a more specific way to say this is that we don’t currently support arbitrary custom logging drivers. There’s implications on the service side with regard to capabilities and placement.

fitzoh commented 4 years ago

There is a workaround for people using EC2 based ECS.

If you set your custom driver as the default for the docker daemon (docs) and don't provide any configuration through ECS, the custom driver will be used (discovered/verified this while trying to test out loki)

hboisgibault commented 2 years ago

Hi @fitzoh, thanks for your help ! Can you share an example for your workaround ?

PrayagS commented 2 years ago

Greetings @fitzoh, thanks for sharing your insights on this. It would go a long way if you can share a detailed example of how you figured this out.

I get the part about not specifying a driver in ECS task definition but didn't quite understand how setting a custom driver for the docker daemon changes configuration on the ECS side. Are you trying to say that you change the docker daemon configuration directly on the host EC2 instances?