[multistage] performance and stability tracker

[walterddr]

We've received many feedback from multi-stage engine beta test. This issue tracks all the reported and fixed performance and stability issues

• [x] create test framework for exact query result matching instead of row-count checker (https://github.com/apache/pinot/pull/9406)
  • [x] follow up, integration test to track all existing pinot queries that we should support (https://github.com/apache/pinot/pull/9412)
• [x] large data block cause grpc failure: https://github.com/apache/pinot/issues/9272
  • [x] follow up, support dataBlock split operation to also be able to split TransferrableBlocks (not only split serialized data, but also split non-serialized data) pipeline the process, especially useful for AGG or SORT that needs to wait for end of stream before returning the first block
• [x] allow pruning on broker: https://github.com/apache/pinot/issues/9165 (fixed partially)
• [x] QueryEnvironment is not threadsafe (https://github.com/apache/pinot/pull/9344)
• [x] Realtime Table is not registering with table cache correctly since getter is hard-coded (https://github.com/apache/pinot/pull/9346)
• [x] limit causes compilation error: https://github.com/apache/pinot/issues/9209 (fixed partially)
  • [ ] follow up, populate limit through stages if possible (e.g. push to table scan node if LEFT JOIN) (#9832)
• [ ] #9626 ((otherwise the mailbox map will not clear mailbox object which potentially can still contain the array buffered data))
• [ ] cancel stuck queries: https://github.com/apache/pinot/issues/9241
• [ ] Pinot component restart didn't automatically make broker picked up server instance changes when it is configured to launch with V2 engine
• [ ] improve resource contention execution failure/timeouts (https://github.com/apache/pinot/issues/9563)

Larger design initiative to improve query runtime

• [x] #9615
• [ ] #9655

[walterddr]

CC @siddharthteotia ^

[siddharthteotia]

Should we separately track resource / thread pool / execution scheduling issue or add it here ?

Without that, this is unlikely to work for a mixed workload deployment where few new (join, subquery) queries go down the new engine (via option) with 2 or more stages whereas existing prod queries continue to run in the current engine with exactly 2 stages.

For users who are purely interested in using the new engine, this is not that big of a problem.

But since there is still a long way to go before fully adopting multi-stage engine for all queries, we need to solve this otherwise it will hinder usage / adoption of new engine's functionality for existing production users.

[walterddr]

i think we should track them together here. I already have some in mind