Auto-restricting scheduler plugin

[JSKenyon]

Yet another alternative to annotation. This is based on comments by @sjperkins in #61.

This is probably the most powerful incarnation yet - it requires zero changes to existing code bar the installation of the plugin. Somewhat distressingly, this renders annotation somewhat pointless as the restrictions can be set directly without annotating layers/tasks. This makes for a lightweight approach that is largely problem agnostic (though there is still a huge amount of room for improvement).

[JSKenyon]

For anyone interested, this is a screen shot of the dashboard without installing the plugin (default): And this is a screenshot with the plugin installed: Red corresponds to transfers between nodes - this is what we want to avoid.

[sjperkins]

Very cool! Keen to drill into this more next week. I am not distressed by the lack of annotations -- they will have their uses, but obviously the fewer needed the better!

[sjperkins]

I think annotating root and terminal layers and feeding that apriori info into this strategy might be useful for some non-trivial distributed cases.

[JSKenyon]

Interestingly, this now works even with optimization enabled. Pretty cool side effect. I agree that this is nowhere near ready for every case. As you say, maybe annotations become a way to guide a more generic scheduler plugin in complicated cases.

[sjperkins]

Interestingly, this now works even with optimization enabled. Pretty cool side effect.

There are no annotated layers, so no annotations are sent to the scheduler (or processed by the plugins).

I suspect this will break if annotations are re-introduced.

[JSKenyon]

I suspect this will break if annotations are re-introduced.

Of course. But cool in the interim.

[JSKenyon]

I am going to press the button. Known caveats:

• Plugin is not perfect but seems to do the correct thing in many common use-cases.
• There is an awkward use of delayed in executor.py to coerce dask-ms coordinate writes into the graph (rather than being isolated nodes). This is a kludge as they otherwise confuse the plugin. This will need to be improved.
• Scheduling tests have been commented out for now. These will be re-added later, but the nature of the plugin makes it difficult to test.