Split single compose to disjoint task definitions, so they can be run/scaled independently

[davemkirk]

I see this as a comment in amazon-ecs-cli/ecs-cli/modules/compose/cli/ecs/app/command.go

// Note: A project is scoped to a single compose yaml with multiple containers defined
// and today, 1 compose.yml has 1:1 mapping with a task definition.
// TODO: Split single compose to disjoint task definitions, so they can be run/scaled independently

This would help me a lot and seems more consistent with how I am able to use docker-compose against swarm or even a single local host.

Is this still a goal on the roadmap for ecs-cli? Let me know if there is anything to do to help with this.

[rprieto]

The independent scaling is very important for us as well. This will be the difference between using ecs-cli for testing purposes (1 instance of everything), and being a tool we can use to orchestrate production deployments (where each container needs to be scaled separately).

The cli could eventually provide something like ecs-cli compose --project-name myproject service scale foobar=2, but for now we'd be happy to manage scaling separately using the raw aws ecs CLI, which is possible as long as every container becomes a separate "service".

One tricky area might be knowing which containers to terminate, for example if 2 projects share the same cluster, or if some services within the cluster were created outside of ecs-cli. Maybe this could be based on naming conventions, e.g. ecscompose-$project-$service.

[errordeveloper]

We have found this behaviour quite surprising when trying to use a compose file with about a dozen services for our microservices reference app.

cc @bchav

[errordeveloper]

I've looked at the code, and it seems like this a very fundamental assumption and it'd would require significant amount of work to fix, and unfortunately the code is not very Go-idiomatic and would take me a long time to wrap my head around... I'd wish I could figure out a simple fix.

[errordeveloper]

I'd wish I could figure out a simple fix.

I mean, a fix in Go code that I could submit as a PR, not a work-around (we have already figured that splitting the file is what we will have to do).

[macrael]

If you split the file, can you still run your dev environment nicely?

[FernandoMiguel]

anyone found a way given a single docker-compose.yml file to create multiple services?

[eulersson]

Apart from scaling I guess we would also benefit from the service discovery features in docker compose? Like... my (docker) service named "backend" can be accessed from the "frontend" one with http://backend. I do that a lot, and for development it's so great, and if I split the files into different tasks I don't think I could keep doing that.

I am still trying to find the most efficient workflow, how do you guys do it?

[david-house-harvard]

@docwhite It seems like ECS Service Discovery is what you're looking for. I haven't quite gotten it to work yet, but in my reading about this issue documentation seems to suggest that you need to do some legwork in order to recreate the functionality provided by docker-compose's bridge network. Alternatively, if you don't want to bother with Fargate, you could aways run ECS with docker's bridge network and I think you'd achieve similar results.

Service DNS routing aside, I agree with other commenters in the desire for ECS's task and service abstraction to more closely match docker-compose's service abstraction. At the moment, ecs-cli has limited utility for me because of the gerrymandering you have to do with your docker-compose.yml files.

[eulersson]

@david-house-harvard Thanks a lot for your reply! Yes... In the end I went down that path. I wrote about each step in detail if you are curious:

https://hackernoon.com/microservices-on-fargate-part1-f26a318827a8 https://hackernoon.com/microservices-on-fargate-part2-f29c6d4d708f

[hackel]

I honestly can't imagine a use-case where this tool would be useful for anything without this feature. It just makes no sense to run entirely separate containers in a single task/service. Yet there has been no progress on this issue in over 3 years. What problem is this tool even attempting to solve, if you can't take an existing docker-compose.yml file and run it without extensive modification, file splitting, etc.?

[eulersson]

Scaling services independently. Think of the classic full stack example where if you have all your containers together into one task you cannot scale them independently...

On 17 Jul 2019, at 17:45, Ryan Hayle notifications@github.com wrote:

I honestly can't imagine a use-case where this tool would be useful for anything without this feature. It just makes no sense to run entirely separate containers in a single task/service. Yet there has been no progress on this issue in over 3 years. What problem is this tool even attempting to solve, if you can't take an existing docker-compose.yml file and run it without extensive modification, file splitting, etc.?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ghost]

Any plans to implement this? This seemed like a great tool until I discovered this TODO.

[ttj4]

I can still see this as TODO in the latest master branch as well! 4+ years and still counting.