thelastpickle / cassandra-reaper

Automated Repair Awesomeness for Apache Cassandra
http://cassandra-reaper.io/
Apache License 2.0
481 stars 216 forks source link

Ignore repair_run_by_cluster_v2 rows with no corresponding repair #1478

Closed SesquipedalianDefenestrator closed 3 months ago

SesquipedalianDefenestrator commented 3 months ago

When bad things (like Reaper filling the drive with logs) happen, it's possible to end up with repair_run_by_cluster_v2 rows with no corresponding repair row, which breaks Reaper. So, just skip them.

github-actions[bot] commented 3 months ago

No linked issues found. Please add the corresponding issues in the pull request description.
Use GitHub automation to close the issue when a PR is merged

SesquipedalianDefenestrator commented 3 months ago

I'm not actually sure if anything will clean up the repair_run_by_cluster_v2 rows, but I'm not sure it's a problem. The mismatch between repair_run and repair_run_by_cluster_v2 is frequent enough to be a pain (basically, a percentage of the time when anything goes wrong with the cluster and Reaper logs until the drive fills), but not often enough to generate measurable load.

SesquipedalianDefenestrator commented 3 months ago

Looks like there's an issue for this already as well: https://github.com/thelastpickle/cassandra-reaper/issues/1463

Miles-Garnsey commented 3 months ago

I'm looking at this, having some issues with my local setup as far as testing goes so bear with me while I try to work around them.