search

dask / dask-cloudprovider

Cloud provider cluster managers for Dask. Supports AWS, Google Cloud Azure and more...
https://cloudprovider.dask.org
BSD 3-Clause "New" or "Revised" License
130 stars 107 forks source link

Add support for AWS Batch #73

Open AlJohri opened 4 years ago

AlJohri commented 4 years ago

It seems like AWS Batch can handle a lot of the logistics of bringing up and down the cluster automatically given a Docker image. I'm curious if it would make sense to integrate this into cloudprovider in some way: https://docs.aws.amazon.com/batch/latest/userguide/multi-node-parallel-jobs.html

jacobtomlinson commented 4 years ago

I would be supportive of this. It would be good to add a BatchCluster as another option.

I don't currently have time to work on this, but would be happy to review a PR.

hammer commented 4 years ago

I wonder if there are any lessons to be learned or integration opportunities with https://github.com/nrdg/cloudknot?

jacobtomlinson commented 4 years ago

Thanks for highlighting this @hammer.

I'm not sure the two are trying to achieve the same goals. Cloudknot seems to be trying to execute functions remotely on Batch from within some Python code.

Dask would be attempting to bootstrap a distributed cluster on Batch and then run the function on the distributed cluster.

samuelsinayoko commented 4 years ago

Hi! Haven't contributed to Dask before but I'd be willing to look into this issue. I'm guessing the idea is to write a class similar to the Fargate one in https://github.com/dask/dask-cloudprovider/blob/master/dask_cloudprovider/providers/aws/ecs.py#L1167 but targeting AWS Batch?

jacobtomlinson commented 4 years ago

@samuelsinayoko that's right!

imanebosch commented 3 years ago

Any news?

vkarunarthne commented 1 year ago

This would be great feature if supported