eirrgang
commented
4 years ago Responding to https://github.com/SCALE-MS/scale-ms/issues/11#issuecomment-578200285
Whether or not it is a technical constraint, it makes sense to assume that a "worker" maps to an ensemble member, and it simplifies the current conversation.
Then
requirement: the execution layer needs to have a concept of logical groups of workers
requirement: the framework needs a scoping concept in which ensemble members are easily mapped to workers within a logical group
If workers can communicate directly with each other, that is probably a special case that would enable some great optimizations, but which we don't need to consider right this moment. The more general requirements are that
- the worker can remain active (but idle, possibly for as much as a few seconds) while waiting for data or instructions but can be told to stop waiting and exit.
- the framework can receive and place data across the ensemble while the workers are active
In order for the execution manager to decide whether to keep workers active or not, it is probably very important to be able to distinguish between an ensemble (or subunit) that is synchronized. It is sufficient, but not necessary, to be able to synchronize launch of workers in an ensemble. Perhaps the necessary condition is to be able to assert or identify workers entering a synchronized ensemble scope.
Note, though, that we are considering use cases that do not require strong synchronization, but where we can benefit from trading the wasted time of relaunching ensemble workers for the wasted time of waiting for data on active workers. We probably need to articulate that concisely, somewhere, and describe the scenarios or limiting parameters governing the effectiveness of the trade-off, but the targeted case is where the number of ensemble members is evenly divisible by the number of co-executing workers, and where the ensemble coupling interval is on the order of ~10 minutes or less, such that simulation tasks launched at the same time don't finish more than a few seconds apart. The most exciting use cases are methods that require or benefit from much shorter coupling intervals across larger ensembles.
Input and output data may be scoped spatially or temporally.
Spatially: Graph topology or outer data dimensions allow slices of data to be referenced without creating a data dependency across an entire ensemble of operations.
Temporally: groups of operation nodes, or subgraphs, are directly analogous to scoped program structures, and analogous data scoping semantics can be used to support data locality optimizations for sequences of tasks associated with a pipeline in an ensemble work flow, including tasks that are adaptively added after the method begins executing.
The goal with this issue is to demonstrate the abstract and concrete graph management for
Update: @mrshirts has pointed out that there are a lot of adaptive use cases that warrant additional consideration for iteration functionality that is optionally unordered. Some initial ideas are reflected in PR #28