BrianGallew / cassandra_range_repair

python script to repair the primary range of a node in N discrete steps
MIT License
109 stars 47 forks source link

critical bugfix: fixed node range assignment #18

Closed pauloricardomg closed 9 years ago

pauloricardomg commented 9 years ago

After deploying the script in my system I was quite impressed to see that cluster-wide repairs that took many hours before were taking only a few minutes.

In order to investigate this, I manually ran nodetool repair -par -pr ks cf in one of the nodes (with token 133971098536869478390145977979120800266) and got the following token range for that node: (127605887595351923798765477786913079296,133971098536869478390145977979120800266) The execution took about 6 minutes for that particular column family.

Then I went and executed range_repair.py -par -k ks -c cf -s 1 -v -d, and was surprised to find out that the script considered the following token range for that node: (133971098536869478390145977979120800266, 134695103572871475120919115441741583695) The execution took a few seconds, which was quite strange.

When observing the generated range by the script, and the actual node range (as generated by nodetool), I quickly realized the script was considering a node's range the region between its token and the NEXT node's token, when in reality a node's range is the region between its token and the PREVIOUS node's token.

This PR fixes this behavior. If what I said above is correct (which I believe it is), the script never worked before.

You can confirm that a node's range is comprised by the region of the ring between itself and its predecessor by checking the following cassandra docs:

rcoli-evb commented 9 years ago

If it's any consolation, I never, ever get this right the first time. When speaking about Cassandra, I generally instruct the audience that it doesn't matter conceptually, except in cases like this.. :D