Open tokenrove opened 1 year ago
Discussed on Slack, but my guess is that this is INTERSECT
which results in a cloning of its left input, rather than a Let
/Get
binding. There are 266 occurrences of INTERSECT
, and .. to the extent that the tree of them has a deep first input that will grow exponentially.
A plausibly easy fix is to introduce a Let
binding at the moment of INTERSECT
planning, and to create and use a Get
node for the two uses of it. This .. has the potential to cascade into unknown horror because it changes the plan structure, but it could actually improve some optimization as the common sub-expression gets pre-identified for it (or the other direction; we'll see).
This is a bug in sql/src/plan/query.rs
in how it handles SetOperator::Intersect
planning. In particular, it doesn't create a Let
binding and clones the lhs, resulting in an exponential memory increase in cases like the above.
What version of Materialize are you using?
v0.40.0-dev (919d05bd8)
How did you install Materialize?
Built from source
What is the issue?
The query included below seems to take an extremely long time to plan. perf suggests that it spends almost all its time in
HirRelationExpr::typ()
, underplan_query
.(apologies for the size and formatting, but sqlformat had a hard time with it, and I don't have reduction implemented yet in the fuzzing I'm doing.)
Relevant log output
No response