MaterializeInc / materialize

The data warehouse for operational workloads.
https://materialize.com
Other
5.66k stars 457 forks source link

[Epic] Variadic outer-join uncertainty reduction #25340

Open lfest opened 4 months ago

lfest commented 4 months ago

Feature request

There is a lot of uncertainty around the variadic outer-join optimization that makes it not possible to estimate the time required with any credibility.

  1. The code on the MVP feature branch is known not to be correct
  2. There is no understanding of how it would interplay with other (important) optimizations
  3. There is no sense of which clients, if any, it could apply to
  4. As with all highly-speculative optimizations, it might turn out to cause regressions

The work in Q1 will be to reduce uncertainty sufficiently to be able to estimate the work involved in completing an implementation. Specifically, read the code and discuss with relevant parties which aspects are likely to be incorrect; identify sections or constructs needing alteration (including in the main branch) to make the candidate implementation correct; define a test plan for verifying not just correctness, but interaction with other optimizations. Use all these to estimate how many weeks will be needed for doing the implementation, testing, and verification, if the investigation doesn't clearly show that this optimization should NOT be done.

### Tasks
- [ ] https://github.com/MaterializeInc/materialize/pull/25502
- [ ] https://github.com/MaterializeInc/materialize/pull/24155
- [ ] https://github.com/MaterializeInc/materialize/pull/24345
- [ ] https://github.com/MaterializeInc/materialize/pull/25591
- [ ] https://github.com/MaterializeInc/materialize/pull/25708
- [ ] https://github.com/MaterializeInc/materialize/pull/26289
- [ ] https://github.com/MaterializeInc/materialize/pull/26357
- [ ] https://github.com/MaterializeInc/materialize/issues/26707
- [ ] Rollout to production
### Follow-ups
- [ ] https://github.com/MaterializeInc/materialize/issues/25918
- [ ] https://github.com/MaterializeInc/materialize/issues/25805
- [ ] https://github.com/MaterializeInc/materialize/pull/26173
- [ ] Support inner joins in `variadic_left.rs`
- [ ] Support local predicates in `variadic_left.rs`
- [ ] Support complex equality predicates in `variaadic_left.rs`
aalexandrov commented 2 months ago

I have a WIP branch with regression tests and for discussing a possible fix here: #26709.

ggevay commented 1 month ago

This seems to be ... done? Even the "Rollout to production" has happened to some degree, right? @mgree , do you remember what is the status here?

mgree commented 1 month ago

I am... not sure. My understanding is it was deployed with a few customers excepted, but I'm not seeing rules for that in LD. 🤔