WallarooLabs / wally

Distributed Stream Processing
https://www.wallaroolabs.com
Apache License 2.0
1.48k stars 68 forks source link

Add coalescing of stateless computations into state steps #455

Open jtfmumm opened 7 years ago

jtfmumm commented 7 years ago

We currently coalesce stateless computations into the same steps since (given that we only have linear pipelines) this leads to less message passing and better performance with correct results. There's no reason in principle why we shouldn't coalesce a series of stateless computations in a preceding state computation, but this functionality is not currently supported.

In order to do this, we would need a way to look up the correct sequence of stateless computations given where the state computation output should be routed to. One possibility is a kind of internal router that maps from target id to Runner-Router chain. To do this, we would have to find a way to assign this target id both to the chain of computations and the router on the step before the state step.

SeanTAllen commented 7 years ago

@jtfmumm do we need to do this before we open source or can it come afterwards?

jtfmumm commented 7 years ago

This would be invisible to the app developer. It's a performance optimization. So I think we can probably open source first.

SeanTAllen commented 7 years ago

@jtfmumm is this done?

jtfmumm commented 7 years ago

No

jtfmumm commented 5 years ago

It's not clear we should do this. See #2633.