search

dask / dask-gateway

A multi-tenant server for securely deploying and managing Dask clusters.
https://gateway.dask.org/
BSD 3-Clause "New" or "Revised" License
137 stars 88 forks source link

Support for AWS ECS Backend #358

Open AndrewHannigan opened 3 years ago

AndrewHannigan commented 3 years ago

Requesting support for AWS ECS with Fargate support. A previous issue was opened with this same request, but closed in favor of Dask CloudProvider. I require Dask Gateway so admins can ultimately control allocation of resource upon request from users, but we like to run everything on ECS with Fargate. ECS support would really streamline the deployment of Dask Gateway for us, as it would not introduce maintenance of a Kubernetes/Hadoop cluster into the mix while still letting admins keep control of resource allocation.

kinghuang commented 3 years ago

Although I closed my request in #317, I'm still interested in having this in the gateway, too.

jcrist commented 3 years ago

I think this would be a fine thing to support, but I'm not likely to work on this myself in the near future. For reference on how one might go about adding this in a PR, see my comment here.

AndrewHannigan commented 3 years ago

Gotcha. @jcrist Does dask gateway use dask cloudprovider at all under the hood for provisioning clusters dynamically? Or did the cloud provider project come after the gateway project? All the functionality that I would need is definitely available in the dask cloudprovider, but I noticed your comments on what a PR might look like didn't mention dask cloudprovider. Just curious if all the work that went into cloudprovider can be leveraged by the gateway.

jcrist commented 3 years ago

Due to the requirements of dask-gateway (DB, resilient to failure, multi-user, scalability issues, etc...), we have to re-implement all backends and don't make use of other dask cluster-managers. In practice this isn't too bad. See https://github.com/dask/dask-gateway/issues/153#issuecomment-540042484 for more info.

jacobtomlinson commented 3 years ago

I'm still disappointed this is the case. Given all the work that goes into packages like dask-cloudprovider it's a shame that effort can't be leveraged here.

I would be very interested in reopening the conversation around how we can better align things to build reusable components.

joncourt commented 3 years ago

There is a ServerDisconnected error issue also with a Fargate based EKS deployment... is that related to this issue or should I raise a new one?

jacobtomlinson commented 3 years ago

That is not related to this issue. Here we are discussing a pure ECS backend implementation, not Kubernetes.

@joncourt If you could raise a new issue that would be great!