search

openlookeng / hetu-core

570 stars 423 forks source link

Olk perf improvement #395

Open sumanth43 opened 2 years ago

sumanth43 commented 2 years ago

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

kind task

What does this PR do / why do we need it: This PR improves performance of openLooKeng

  1. Increase OrcReader#MAX_BATCH_SIZE to 8196
  2. Improve estimation of row count from partition samples
  3. Replace BigArrays with primitive ones in BigInt aggregation
  4. Replace row-wise DefaultPagePartitioner with ColumnarPagePartitioner
  5. Prioritize Utilization Execution Policy
  6. Improve CBO estimates for correlated columns
  7. Allowing Creation Of Dynamic Filters from Table Scan without Predicates
  8. Make partial aggregation adaptive
  9. Prevent Dead Locks With Prioritize Utilization Execution Policy
  10. Prune CTE with Self Join present in DownStream

Which issue(s) this PR fixes:

Fixes #396

Special notes for your reviewers:

it-is-a-robot commented 2 years ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sumanth43 To complete the pull request process, please assign haochending You can assign the PR to them by writing /assign @haochending in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/openlookeng/hetu-core/blob/master/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment