deploy_ecs example doesn't work any more due to docker's ecs support retirement

selfscrum commented 10 months ago

Dagster version

1.6.0

What's the issue?

deployment to ecs is not working any longer with docker compose. The following command is not executing, because the ecs subcommand is not active anymore (retired end of last year):

docker context create ecs dagster-ecs

the error message is

"docker context create" requires exactly 1 argument.

I can create a local context but that just leads to a local docker deployment form the created ECR images.

What did you expect to happen?

Please create a working example for deployment to ECS, with docker compose, AWS Cloud Formation, or terraform.

How to reproduce?

Just follow the example steps.

Deployment type

Docker Compose

Deployment details

Docker Version 24.0.7

Additional information

I tried alternatively to use ecs-compose-x but failed while building the stack, especially since user_code is not accepted as a valid name by cloud formation. This might be an ecs-compose-x issue but keeps me from easily replacing docker compose.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

alex-antonison commented 9 months ago

Are there any plans address this issue?

sbquinlan commented 9 months ago

Yes, we are aware of this and tracking it. We're hoping to prioritize updating the ECS example in the next few months.

ilkkapeltola commented 9 months ago

darn... just when I needed this thing 😬 In the mean time, is AWS copilot a good place to look into or can I get everything I need done in just terraform?

selfscrum commented 9 months ago

You can start with the existing ecs example to docker build and push to ECR. From then on it's all possible in terraform but you have to build the whole infrastructure especially when you want to have a distributed system. It took me some days to puzzle it all together:

ECS/Fargate Services and tasks
Load Balancer
RDS Postgres Database
IAM Roles and Policies
Cloud Map / Servicediscovery and Namespace
CloudWatch Logs But finally I got it going. If time permits I will publish what I learned, I had some dark hours 😅 I especially liked the idea of an S3 bucket for dagster and workspace.yaml which get pushed on the fargate server with a sidecar container so you dont have to rebuild the whole setup on changes...

ilkkapeltola commented 9 months ago

There seems to be a 3-year old terraform module here. That could still be a good starting point I guess. 🤷‍♀️

alex-antonison commented 9 months ago

I looked at that as well and was planning to use that as a starting point. I am curious how much has changed since it is three years out of date. It looks like there are a couple of us interested in this approach so could be worthwhile to start an "open source" and collaborate around it. I also have a colleague who I know has done this as well and plan on reaching out to him.

That being said, curious what direction dagster plans to go since I wouldn't want to invest a bunch of effort into something if they were to come out with a better supported solution which would remove the need to spin up a an open sourced version of deploying dagster to ECS.

selfscrum commented 9 months ago

Here's my quickly patched copy of a project I am currently working on. As stated in the README, its untested in this variant, I had it originally on gitlab. Happy to see someone continuing the work and sends back some merge requests. Good luck!

https://github.com/selfscrum/terraform-aws-dagster-ecs

ilkkapeltola commented 9 months ago

FYI @selfscrum https://github.com/dagster-io/dagster/discussions/14946#discussioncomment-8529552

kristianeschenburg commented 3 weeks ago

@sbquinlan -- have there been any changes on this front?

dagster-io / dagster