search

MaterializeInc / materialize

The Cloud Operational Data Store: use SQL to transform, deliver, and act on fast-changing data.
https://materialize.com
Other
5.72k stars 466 forks source link

Protect against obvious memory leaks #7965

Open benesch opened 3 years ago

benesch commented 3 years ago

We need to find a way to protect against bugs like #7934. One possible test is a release load test that does nothing and asserts that Materialize's memory usage is sane. @ruchirK had some other clever ideas too, like a soft assertion that checks for long running transactions.

cc @mjibson @umanwizard

ruchirK commented 3 years ago

https://github.com/backtrace-labs/poireau was designed for this exact problem btw

maddyblue commented 3 years ago

Could we add since to mz_materialization_frontiers, and assert that it keeps up? That's only tests memory leaks in a specific part of the system, but would have caught this.

ruchirK commented 3 years ago

you would need to plumb through the index's compaction window for that approach to work though. otherwise, you might mistakenly think an index is falling behind when instead the compaction was intentionally turned off.

This also gets tricky if someone is altering the compaction window at runtime -- it could be off and then the since is really far back, and then the user alters it to 1ms, and then it takes a while to catch up. Not insurmountable stuff though

maddyblue commented 3 years ago

Can we use the coord's frontier since? Wouldn't have to care about the compaction window at all, just whatever coord thinks is going on. Maybe not a thing because that view isn't populated by coordinator data today?