marsupialtail
commented
1 year ago What we are doing here:
- Added a stage number to each actor
- The actor cannot start until the global stage number has progressed to its assigned stage number
- The global stage number progresses once all the actors in a previous stage have put in results in DST
- Did not yet test fault tolerance behavior of this, should be ok. This is merely changing the execution order of some operators
- Upon fault recovery actors from any stage smaller than current can be relaunched
- this is ok because stages mostly have to deal with lineage. Normal operation sets the lineage, fault tolerance reads this lineage. As long as the lineage is set correctly in normal operation it should be ok.
This is a significant change in how we are doing things in Quokka.
Up to release 0.1.5 we are doing joins with streaming X join where both sides are built, now we choose one side to be build and block until that is done and then stream through the probe. This is more in line with traditional databases.
Might see performance degradations on joins in TPC-H, that's ok. Join order selection not implemented yet asof this PR.