This is not yet ready for merging as it is too unstable - upstream churn is regularly breaking things. Currently, I can get things sort of working with dask==2021.02.0 and distributed==2021.02.0, but those versions don't necessarily have all the required functionality and certainly have some cluster-breaking bugs. I have also pinned numba==0.52.0 as 0.53.0 currently breaks compilation, likely due to dispatch. This is on my to-do list.
Additionally, there is ugly commented/hardcoded code in the executor that needs to be formally incorporated.
@sjperkins much of the original PR is unchanged, but you can have a look as/if/when you have a moment.
Edit: This does work with the current main branches - somehow I have ended up with a bad distributed version.
This is not yet ready for merging as it is too unstable - upstream churn is regularly breaking things. Currently, I can get things sort of working with
dask==2021.02.0
anddistributed==2021.02.0
, but those versions don't necessarily have all the required functionality and certainly have some cluster-breaking bugs. I have also pinnednumba==0.52.0
as0.53.0
currently breaks compilation, likely due to dispatch. This is on my to-do list.Additionally, there is ugly commented/hardcoded code in the executor that needs to be formally incorporated.
@sjperkins much of the original PR is unchanged, but you can have a look as/if/when you have a moment.
Edit: This does work with the current
main
branches - somehow I have ended up with a baddistributed
version.