Open vmarcos opened 1 year ago
@petrosagg help me validate the impact - would this impact kafka APPEND ONLY sources? I would assume it doesn't given the only subsource is the progress subsource, but wanted to validate. Also for postgres given that none of those should be monotonic that shouldn't be an issue either correct?
@dseisun-materialize Your understanding is correct, neither kafka nor postgres are impacted because the former doesn't have subsources and the latter is not monotonic. We do have loadgen sources however that produce append-only subsources and those are impacted
@vmarcos for my edification what's the impact of not being able to detect if something is monotonic? I'm guessing we can't perform certain optimizations?
@dseisun-materialize Yes, missing out on optimizations is the main impact. Especially for patterns like min/max/top-k, we'd need to build by default large stacks of arrangements (potentially allocating a ton more memory and/or spending more time) for incremental maintenance. After we turn on enable_monotonic_oneshot_selects
in prod, the impact on one-shot SELECT
s of not knowing monotonicity will be smaller, but still not zero.
So I guess that if there are people self-onboarding out there and trying out the auction load generator (because that's the one we point them to for getting started), they could be seeing more expensive plans than they'd need to.
What version of Materialize are you using?
v0.61.0-dev (9cefb13f7)
What is the issue?
The monotonicity information associated to a source should propagate to its subsources, but it presently does not: https://github.com/MaterializeInc/materialize/blob/2fb423ad0dbfd3affa9359088d97170c673e6d5a/src/adapter/src/coord/dataflows.rs#L528-L537
Slack refs: 1, 2.
The issue is problematic with monotonic sources, for which we then fail to produce incremental view maintenance plans taking advantage of monotonicity. To illustrate this behavior, recall that the auction load generator is monotonic. Now, consider the following statements:
The outcome of executing the above statements is:
Above, we see that the
EXPLAIN PHYSICAL PLAN FOR VIEW
variant fails to recognize the monotonicity of the subsources and produce a monotonic incremental view maintenance plan. The plan for a one-shotSELECT
can always exploit monotonicity, by contrast, butmust_consolidate
could in the case of a source that is already monotonic be turned off (see #18732).