Currently when a scheduler goes down and comes back empty all futures on all executors are cancelled/lost.
However, the Executor could choose to hold on to how it produced all futures. This would allow it to resubmit all computations. There are pros and cons to this:
Pro: This would establish the weakest form of full resilience to scheduler failure.
Con: There are two copies of the dask graph lying around. Suddenly the Executor gets more complex.
pelson
commented
7 years ago
Currently when a scheduler goes down and comes back empty all futures on all executors are cancelled/lost.
However, the Executor could choose to hold on to how it produced all futures. This would allow it to resubmit all computations. There are pros and cons to this: