search

Quansight-Labs / dask-jobqueue

Deploy Dask on job schedulers like PBS, SLURM, and SGE
https://jobqueue.dask.org
BSD 3-Clause "New" or "Revised" License
0 stars 1 forks source link

Create a AWS Batch JobQueue Runner #2

Open jerowe opened 4 years ago

jerowe commented 4 years ago

This is fairly self explanatory. We need to extend the base scheduler to implement an AWSBatch, and it should look very similar to SLURM. (Except for Batch! ;-) )

If you need an environment to test on please see the login credentials in Slack. There is a fully functioning AWSBatch cluster.

Here is a tutorial that includes submitting a job to batch. The backend of Batch is ECS, but its supposed to act like a HPC cluster.

Is there a potential gotcha that it doesn't supply IP addresses?

jerowe commented 4 years ago

@prasunanand @dharhas brought up a good point that AWSBatch is built on ECS, so a wrapper, if one is even needed, would fit in more with the dask-cloudprovider repo. I think its still a good to have, so feel free to hop onto the AWSBatch cluster and start investigating. Just keep in mind that the implementation is more along the lines of ECS rather than SLURM.

dharhas commented 4 years ago

There is already AWS Fargate wrapper to run Dask on AWS easily and quickly. Not sure that building this should be a priority.

prasunanand commented 4 years ago

Ah! I have been working on AWSBatch. Code can be found here: https://github.com/prasunanand/dask-jobqueue/commit/c3f7fbf869fe696e2de5134069213c406a07cfbe