matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.77k stars 274 forks source link

[Tech Request]: pipeline refactoring, don't break pipelines if possible #16542

Open badboynt1 opened 3 months ago

badboynt1 commented 3 months ago

Is there an existing issue for the same tech request?

Does this tech request not affect user experience?

What would you like to be added ?

pipeline重构,任何情况下,能不打断pipeline就不要打断。

特别是对于优化器判定为tp的query,需要保证
1. 一定在单cn,单并发执行。
2. 不产生任何额外的goroutine
3. pipeline不断开,不使用channel传递batch
4. 不出现connector,merge,dispatch等算子,必要时使用一个新的exchange算子来连接pipeline。exchange算子直接调用对应scope的root operator call,返回batch
5  对于某些特定算子的优化,例如mergesort,mergegroup,mergelimit,以及join等等

Why is this needed ?

No response

Additional information

No response

fengttt commented 1 month ago

https://github.com/matrixorigin/matrixone/commit/c37e5be9e6e37c29baa9b739b98bf6befbdaad50

fengttt commented 1 month ago

17778 is reverted by #17787