feat(hashjoin): Add fast row size estimation for hash probe

facebookincubator / velox

A composable and fully extensible C++ execution engine library for data management systems.

https://velox-lib.io/

Apache License 2.0

3.54k stars 1.17k forks source link

Closed tanjialiang closed 5 days ago

tanjialiang commented 1 week ago

Add column stats for row container to collect aggregated column stats. The aggregated column stats will be used in hash probe to decide if row size estimation is applicable. If it is applicable, column stats will be used to compose a fast row size estimation to avoid memory exploding when probing and listing results. This added feature makes hash join more performant, and in some extreme skew cases that we've seen in Meta internal queries, it helped to decrease the query latency by >20x.
The work of this feature also helped to discovered a bug in HashTable when using simd for fast path result listing -> when max number of rows is smaller than kWidth, the unsigned integer overflow bug will make the max number of rows be ignored. Fixed the bug and the new test covers that case.

netlify[bot] commented 1 week ago

Name	Link
Latest commit	07ede692e9a2243a7f9aea6a357893d8453e7e87
Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/6740f6ecd8c3000008da482b

facebook-github-bot commented 1 week ago

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 1 week ago

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 1 week ago

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 1 week ago

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 1 week ago

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

zhli1142015 commented 1 week ago

tanjialiang commented 6 days ago

ot successful

I'll do a followup PR for that, to make sure there's no regression if enabled by default.

facebook-github-bot commented 6 days ago

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 6 days ago

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 6 days ago

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 5 days ago

This pull request was exported from Phabricator. Differential Revision: D66064300

facebook-github-bot commented 5 days ago

@tanjialiang merged this pull request in facebookincubator/velox@059337fca8170c2b361ea9d89d6c2cdd9e157c4a.

conbench-facebook[bot] commented 5 days ago

Conbench analyzed the 1 benchmark run on commit 059337fc.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.