search

dask / dask-jobqueue

Deploy Dask on job schedulers like PBS, SLURM, and SGE
https://jobqueue.dask.org
BSD 3-Clause "New" or "Revised" License
234 stars 142 forks source link

Libsubmit #182

Closed dgasmith closed 5 years ago

dgasmith commented 5 years ago

Searching for alternatives to wrap cluster schedulers I ran across libsubmit from the Parsl project. It may be too Parsl-specific for this project, but could provide nice templates for things like Cobalt/Jetstream/etc that are not yet here. I thought it would be good to at least be aware of this project.

guillaumeeb commented 5 years ago

Thanks for sharing @dgasmith, that is really interesting. And I mean that for both the Parsl project and the libsubmit part.

When seeing #49, or also https://github.com/jupyterhub/batchspawner, that's a lot of libraries implementing the same kind of interfaces with jobqueue systems. Libsubmit is the first I see with a goal of extracting only the "submit anything" part, and could maybe be used here. Taking a quick look, it seems overkill and as you said, maybe too Parsl-specific yet, but having such a library would be a great help for many projects.

And yes, at least we can benefit from the experience of dealing with other Jobqueue systems such as Condor, cobalt or others, so thanks again!

Just out of curiosity, do you use Parsl? I can see it's not the same purpose as Dask, but there's some similarities, especially with Dask Delayed, did you compare the two softwares somehow?

dgasmith commented 5 years ago

I do use Parsl and Dask, but just as a flat queue to ship jobs out to compute nodes rather than full blown workflows. So I cannot comment too much on the more specific features of the libraries.

I agree that having a central place for spawning tech would be ideal. I have a close enough association with the Jupiter and Parsl folks that I could likely setup a conversation if desired.

guillaumeeb commented 5 years ago

I do use Parsl and Dask, but just as a flat queue to ship jobs out to compute nodes rather than full blown workflows

So why are you using both libraries? It's sound like Parsl would be more adapted to this use case?

I could likely setup a conversation if desired

Currently I don't see the need for that, but maybe in the future! Thanks.

dgasmith commented 5 years ago

We have a database and workflows project for the quantum chemistry and treat Dask/Parsl/Balsam/Radical/BOINC/etc as general distributed computing libraries while rarely leveraging their workflow components. Users are free to pick the distributed computing project that they know and matches their use case.

Something like Dask-jobqueue will allow Dask users to more easily access supercomputing resources that, granted, is similar to Parsl's functionality.

guillaumeeb commented 5 years ago

Thanks for this information @dgasmith, closing for now.