Open jtfmumm opened 7 years ago
@jtfmumm do we need to do this before we open source or can it come afterwards?
This would be invisible to the app developer. It's a performance optimization. So I think we can probably open source first.
@jtfmumm is this done?
No
It's not clear we should do this. See #2633.
We currently coalesce stateless computations into the same steps since (given that we only have linear pipelines) this leads to less message passing and better performance with correct results. There's no reason in principle why we shouldn't coalesce a series of stateless computations in a preceding state computation, but this functionality is not currently supported.
In order to do this, we would need a way to look up the correct sequence of stateless computations given where the state computation output should be routed to. One possibility is a kind of internal router that maps from target id to Runner-Router chain. To do this, we would have to find a way to assign this target id both to the chain of computations and the router on the step before the state step.