This library provides some useful decorators for dask_jobqueue. It also expands it's scope to include MPI workloads, including extending configuration options for such workloads and heterogeneous resources.
I have seen failures in the scheduled CI (see this for an example) that seems to occur pretty regularly but is not linked to any particular iteration in the CI (though seems to occur more frequently with PBS).
Since I'm shooting in the dark, I just added something that shouldn't do any harm but might solve the problem. CI is passing so I'm going to merge and wait and see if that resolves the issue going forward.
I have seen failures in the scheduled CI (see this for an example) that seems to occur pretty regularly but is not linked to any particular iteration in the CI (though seems to occur more frequently with PBS).
Since I'm shooting in the dark, I just added something that shouldn't do any harm but might solve the problem. CI is passing so I'm going to merge and wait and see if that resolves the issue going forward.