Add support for JOIN - Multi-stage execution query engine

[walterddr]

We have come up with a design doc to support complex multi-stage query execution.

Motivation

Today's Pinot query engine only supports a single scatter-gather model. This design runs very well if most of the heavy-lifting computation can be push down to segment servers, however it also have some limitations we would like to address. Namely,

• Remove bottleneck in Pinot Broker when heavy reduce is needed.
• Support SQL features such as window functions, distributed join, etc.
• Support standardized full SQL semantics

Proposal

We propose to build a new multi-stage query engine to address these, specifically

• a new SQL planner that can support full-SQL semantic planning.
• a new execution engine that can support multi-stage, DAG-based execution.

Design Doc

Please see https://docs.google.com/document/d/10-vL_bUrI-Pi2oYudWyUlQl9Kf0cLrW-Z8hGczkCPik/edit# for more details.

[yupeng9]

This is so exciting!

Strong +1 to this proposal. It'll shape Pinot into the next-generation architecture and unlock many additional capabilities. For example, from the Uber side, this could help us extend the current lookup join and explore other join strategies such as broadcast join and colocated join.

It'll be great if you can push your prototype to a feature branch, so we can review and contribute over that.

[kishoreg]

The most awaited feature!

[siddharthteotia]

+1 on this

[xiangfu0]

+1000！

[mayankshriv]

Lets knock this down. +1

With all the +1's above, and no -1, I think it is safe to conclude that we can move ahead with detailed design+impl phase @walterddr.

[siddharthteotia]

Linking related old issue for reference - https://github.com/apache/pinot/issues/5664

This one should be used and the 5664 can remain open or be closed

[mcvsubbu]

@walterddr can you please open up access to the design doc?

[siddharthteotia]

First PR for query planner merged to feature branch https://github.com/apache/pinot/pull/8340/