Open benesch opened 3 years ago
https://github.com/backtrace-labs/poireau was designed for this exact problem btw
Could we add since
to mz_materialization_frontiers
, and assert that it keeps up? That's only tests memory leaks in a specific part of the system, but would have caught this.
you would need to plumb through the index's compaction window for that approach to work though. otherwise, you might mistakenly think an index is falling behind when instead the compaction was intentionally turned off.
This also gets tricky if someone is altering the compaction window at runtime -- it could be off and then the since is really far back, and then the user alters it to 1ms, and then it takes a while to catch up. Not insurmountable stuff though
Can we use the coord's frontier since? Wouldn't have to care about the compaction window at all, just whatever coord thinks is going on. Maybe not a thing because that view isn't populated by coordinator data today?
We need to find a way to protect against bugs like #7934. One possible test is a release load test that does nothing and asserts that Materialize's memory usage is sane. @ruchirK had some other clever ideas too, like a soft assertion that checks for long running transactions.
cc @mjibson @umanwizard