Open albsch opened 4 years ago
On the branch random_slicing we developed a prototype of an integration of Random Slicing into RCL. With the prototype we tried to replace Consistent Hashing with Random Slicing without affecting the system structure too much. From this prototype the following learnings resulted:
Overall we assess a restructuring of RCL to replace Consistent Hashing with Random Slicing as beneficial for the system. However, the amount of necessary work and sources for potential errors to achieve this is significant.
Now that the code base is much smaller, possible changes in the architecture could be discussed. Possible starting points mentioned by Scott Lystig Fritchie:
[ ] The only string-to-integer hash algorithm is SHA-1.
[ ] The hash “rings” integer interval is the range 0 to 2^160-1.
[ ] The number of partitions is fixed.
[ ] The number of partitions must be a power of 2.
[ ] The size of each partition is fixed.
[ ] Historically, the “claim assignment” algorithm used to assign servers to intervals on the ring were buggy and naive and frequently created imbalanced workload across nodes.
[ ] Server capacity adjustment by “weighting” did not exist: all servers were assumed equal.
[ ] No support for segregating extremely “hot” keys, for example, Twitter’s “Justin Bieber” account.
[ ] No effective support for “rack-aware” or “fault domain aware” replica placement policy
Any of these changes will only concern the next major release most probably.