This offers an alternative to #58. This approach is fully automated which means it can be applied with few to no changes to the existing code. Initial tests have shown that this can completely eliminate transfers between workers (subject to some conditions).
@sjperkins I would appreciate any thoughts you may have on improving/extending this approach. The changes which will be of particular interest to you are the those to executor.py and scheduling.
Known issues:
Disconnected nodes will be assigned their own abstract group which may be problematic.
Currently, each group is assigned to a specific worker. Some groups may be too small/too large for this simplistic approach.
The current code is not suitable for reannotation as the state of the graph determines how groups are assigned. The annotation code should first delete existing "__group__" annotatations before reannotating.
JSKenyon
commented
3 years ago
This offers an alternative to #58. This approach is fully automated which means it can be applied with few to no changes to the existing code. Initial tests have shown that this can completely eliminate transfers between workers (subject to some conditions).
@sjperkins I would appreciate any thoughts you may have on improving/extending this approach. The changes which will be of particular interest to you are the those to
executor.pyandscheduling.Known issues:
The current code is not suitable for reannotation as the state of the graph determines how groups are assigned. The annotation code should first delete existing"__group__"annotatations before reannotating.