[core] Optimize IncrementalStartingScanner to randomlyExecute

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

https://paimon.apache.org/

Apache License 2.0

2.46k stars 969 forks source link

Closed JingsongLi closed 2 months ago

JingsongLi commented 2 months ago

Sequence batch should not be used; it only optimizes memory, and we only need to return the results in order.

Randomly execution can improve performance.

wwj6591812 commented 2 months ago