Use `dask` to launch cluster jobs

[elcorto]

Could be used to replace the custom template file workflow. Neede packages:

dask
dask-jobqueue
distributed

[elcorto]

If this works, we can rename run_local() back to run() and just pass a dask.distributed.Client and use results=client.gather(client.map(worker, params)) instead of results=pool.map(worker, params). Then run() will start batch jobs. The idea behind the name run_local() was to later have a run_cluster() function that would implement a cluster workflow using prep_batch() etc, which would now be unnecessary, since run() can do both.

[elcorto]

This is being worked on in branch https://github.com/elcorto/psweep/tree/feature-dask. Expect frequent rebase and force-push, this is a feature branch :)

So far we have test_dask_local_cluster which works.

Todos

• [x] split batch_with_git example into batch_templates (w/o git) and batch_templates_with_git
  • [x] make sure that test_all.py::test_run_all_examples pickes up all of them
• [x] Add small example using SLURMCluster (for now in the manual)
• [x] Add example explaining all parameters of SLURMCluster
• [x] Update manual (partially done)
• [x] CI: install dask in addition to test reqs

[elcorto]

• examples: https://github.com/elcorto/psweep/tree/feature-dask/examples/batch_dask
• improved docs: https://github.com/elcorto/psweep/blob/feature-dask/doc/source/written/manual.md#running-on-hpc-compute-infrastructure

[elcorto]

Implemented in https://github.com/elcorto/psweep/releases/tag/0.11.0