Open lfest opened 4 months ago
This also kind of seems done? @mgree , do you maybe know what was the plan here?
Absolutely no idea---never saw this epic. 😔
So, the last thing I remember about this is that it was rolled out to some customers (maybe most?), but there weren't big gains, and there were some regressions in some cases. Also, we couldn't roll it out to all customers, because there would have been a huge regression for at least one customer (found by Alex doing experiments with unbilled replicas, if I remember correctly). But I'm not sure what was the plan forward with this.
IIRC the regression was in the combination of EDJ and the variadic outer join lowering. It looks like there are targeting rules to turn EDJ on for two environments, but it's not generally deployed.
I think we concluded that we needed a less eager EDJ, but I sadly don't remember the concrete details of the regression or our proposed fix.
There was a user environment with a 100x regression from just eager delta joins: https://materializeinc.slack.com/archives/C02PPB50ZHS/p1712074981313969?thread_ts=1712073070.254049&cid=C02PPB50ZHS
Validate the eager delta join optimization in specific client settings. This will also incorporate experimentation.