Closed elcorto closed 9 months ago
If this works, we can rename run_local()
back to run()
and just pass a dask.distributed.Client
and use results=client.gather(client.map(worker, params))
instead of results=pool.map(worker, params)
. Then run()
will start batch jobs. The idea behind the name run_local()
was to later have a run_cluster()
function that would implement a cluster workflow using prep_batch()
etc, which would now be unnecessary, since run()
can do both.
This is being worked on in branch https://github.com/elcorto/psweep/tree/feature-dask. Expect frequent rebase and force-push, this is a feature branch :)
So far we have test_dask_local_cluster
which works.
Todos
batch_with_git
example into batch_templates
(w/o git) and batch_templates_with_git
test_all.py::test_run_all_examples
pickes up all of themSLURMCluster
(for now in the manual)SLURMCluster
Implemented in https://github.com/elcorto/psweep/releases/tag/0.11.0
Could be used to replace the custom template file workflow. Neede packages: