Support multi-thread execution on multistage engine - Githubissues

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore

https://pinot.apache.org/

Apache License 2.0

5.41k stars 1.27k forks source link

Support multi-thread execution on multistage engine #9655

Open walterddr opened 1 year ago

walterddr commented 1 year ago

as a continuation of the discussion in #9615 that make the decision to not include multi-thread execution as part of the previous design review.

Mainly discuss the design for how to multi-thread execute in multi-stage engine, 2 initial thoughts

execute operators in pipeline fashion with block boundary.
execute partitioned blocks in parallel within each stage execution.

Discussion in previous threads: https://github.com/apache/pinot/issues/9615#issuecomment-1282532520

walterddr commented 1 year ago

CC @agavra