Closed coastalwhite closed 3 weeks ago
When the optimizer spots a chain of WITH COLUMNS, it tries to pull expressions as close to the source as possible.
I think we must try to pull as close to the first WITH_COLUMNS
. Closer to the source isn't per se beneficial, as the crossing operations like Filter
and Join
make those node more expensive.
Currently, we only optimize sequential WITH_COLUMNS
calls if there is nothing in between them. So at the moment, closer to source is roughly equal to first WITH_COLUMNS
.
I think this CI failure is a false negative
Attention: Patch coverage is 88.84892%
with 31 lines
in your changes are missing coverage. Please review.
Project coverage is 81.41%. Comparing base (
ec08d76
) to head (f9bee6f
). Report is 4 commits behind head on main.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
I think this PR is mostly ready. Some points to specifically look at are.
ProjectionOptions
correctly.AExpr::Wildcard
and AExpr::Function
.Bitmap
into an IndexSet
. Perhaps, it is worth it just to make an index set. That is what the commented Bitset
struct is. But if it is not wanted, I will just remove that before merging.IRBuilder
would require more code. I think it is fine here, but maybe you don't agree.Nice one! I have left a few comments.
Maybe wait with merging this. I still need to add the traversal to check whether the optimization pass is even necessary
Okay! This is ready to get merged :rocket:
Comparing coastalwhite:opt-cluster-with-columns
(f9bee6f) with main
(4375930)
✅ 37
untouched benchmarks
This adds a new optimization pass for the query engine that clusters several sequential
WITH COLUMNS
calls.When the optimizer spots a chain of
WITH COLUMNS
, it tries to pull expressions as close to the source as possible. If all expressions can be pulled up closer to the source, theWITH COLUMNS
node is removed entirely.