reiserm / midtools

Data analysis software to process and analyze XPCS data acquired at European XFEL.
BSD 3-Clause "New" or "Revised" License
1 stars 1 forks source link

Persist dask cluster across compute calls #4

Open RobertRosca opened 4 years ago

RobertRosca commented 4 years ago

Currently each new compute method call restarts the Dask cluster, I'm not too sure what the consequences of this are, but it's possible that this can reduce performance and lead to some of the stability issues that occasionally occur.

At a minimum, it means that dask cannot fully lazily build the task graph, as the cluster gets reset between functions. This is closely linked to issue #1.