MaterializeInc / materialize

The data warehouse for operational workloads.
https://materialize.com
Other
5.66k stars 457 forks source link

[Epic] Eager delta join experimentation and client analysis #25337

Open lfest opened 4 months ago

lfest commented 4 months ago

Validate the eager delta join optimization in specific client settings. This will also incorporate experimentation.

### Tasks
- [ ] https://github.com/MaterializeInc/materialize/pull/25026
- [ ] https://github.com/MaterializeInc/materialize/pull/25473
- [ ] https://github.com/MaterializeInc/cloud/pull/8779
- [x] https://github.com/MaterializeInc/materialize/pull/26308
- [x] Identify prospects based on an `mzexplore` run
- [x] Execute experiments using #25273
- [x] Rollout to production
### Follow-ups
- [ ] Figure out what causes of higher latency when using eager delta joins
ggevay commented 1 month ago

This also kind of seems done? @mgree , do you maybe know what was the plan here?

mgree commented 4 weeks ago

Absolutely no idea---never saw this epic. 😔

ggevay commented 4 weeks ago

So, the last thing I remember about this is that it was rolled out to some customers (maybe most?), but there weren't big gains, and there were some regressions in some cases. Also, we couldn't roll it out to all customers, because there would have been a huge regression for at least one customer (found by Alex doing experiments with unbilled replicas, if I remember correctly). But I'm not sure what was the plan forward with this.

mgree commented 3 weeks ago

IIRC the regression was in the combination of EDJ and the variadic outer join lowering. It looks like there are targeting rules to turn EDJ on for two environments, but it's not generally deployed.

I think we concluded that we needed a less eager EDJ, but I sadly don't remember the concrete details of the regression or our proposed fix.

ggevay commented 3 weeks ago

There was a user environment with a 100x regression from just eager delta joins: https://materializeinc.slack.com/archives/C02PPB50ZHS/p1712074981313969?thread_ts=1712073070.254049&cid=C02PPB50ZHS