Will Dask Developers want the BashWrapper in JobQueue?

[Adam-D-Lewis]

So I'm also new to Open Source Development, and so this issue is mostly so I can understand the process, but I have a few questions.

• Do the developers of dask know that we want to add the BashWrapper to Jobqueue, and do they agree? I guess the alternative would be to make a separate package that depends on Dask JobQueue.
• If not, should we bring it up in an issue and get some feedback first?

[dharhas]

I think there is confusion here in terms of naming.

There are two things.

a. A simple text based format to define a simple DAG that can be converted to a Dask High Level Graph b. a wrapper to make it easier to call bash scripts and command line programs.

I think (a) is worth raising on the Dask issue tracker (it is orthogonal to dask_jobqueue) for ideas.

I'm not sure (b) is worth spending much time on, at least not at this point. A simple wrapper around subprocess.run will work for most cases.

Calling it a BashWrapper will cause confusion with upstream developers.

[jerowe]

There is definitely some question of where things belong conceptually, and how to use Dask best as opposed to hammering it into acting like an HPC scheduler.

I think we should continue the work here, and either add it to Dask JobQueue or create a separate Dask Cookbook Repo (and maybe even a separate Dask NextStrain Cookbook since that's the target for now) that has different recipes on how you would approach using Dask in an optimal way for various scenarios.

This also relates to a post by a JobQueue maintainer here - https://discourse.covid-oss-help.org/t/helping-nextstrain/284/24, which feeds back into the issue of not hammering a round Dask into a square HPC hole. ;-)