search

thelastpickle / cassandra-reaper

Automated Repair Awesomeness for Apache Cassandra
http://cassandra-reaper.io/
Apache License 2.0
490 stars 218 forks source link

auto determine segment count #280

Open rustyrazorblade opened 7 years ago

rustyrazorblade commented 7 years ago

Project board link

A suggestion from the excellent Chris Lohfink on IRC:

fwiw on limiting overstreaming, a nice improvement would be to use the size_estimates system table to split based on number of partitions instead of a fixed value across the board

┆Issue is synchronized with this Jira Story by Unito ┆Issue Number: REAP-180

adejanovski commented 7 years ago

From #279 by @eriksw :

For keyspaces with a few extremely large CFs and a large number of comparably microscopic CFs, the time to do a whole-keyspace repair with a large number of segments (needed to make progress within acceptable timeouts on the big CFs) is nontrivially impacted by the time that gets spent coordinating the tiny-range repairs of each of the smaller CFs. Manually creating a couple different runs, one for each of the very large CFs and one for the enumerated small CFs works around this, but it's an annoying maintenance burden to have to re-do the schedules in reaper every time a CF is added or dropped. It'd be nice if reaper enumerated CFs and their sizes when generating segments and made it so that each segment of the repair covers (by rough estimates) the same number of bytes, whether that's a very tiny range in a single large CF or -pr-equivalent of a large number of small CFs.