Can we optimise unordered set rehashing process ?

[chris-nee]

Some notes that may be relevant - https://en.wikipedia.org/wiki/Consistent_hashing

[benedictkao]

I read up a bit and watched some videos, and looks like this is primarily used for load balancing in a multi-server BE. Not sure if the performance tradeoffs are worth it. This will require more investigation

[benedictkao]

Advantages:

• Load balancing: Consistent hashing helps to distribute data evenly across nodes, which can improve load balancing and prevent overloading of individual nodes.
• Fault tolerance: Consistent hashing provides a mechanism for handling node failures by redistributing data to other nodes. This helps to ensure that the system remains available and responsive even in the presence of failures.
• Scalability: Consistent hashing allows for the addition and removal of nodes without having to rehash all the data, which can be time-consuming and resource-intensive. This makes it easier to scale the system as the number of nodes and the amount of data grows.

Disadvantages:

• Uneven data distribution: Although consistent hashing aims to distribute data evenly across nodes, it is still possible for data to be unevenly distributed in practice. This can result in some nodes being overloaded with data while others are underutilized.
• Performance impact: Consistent hashing can have a performance impact, particularly in large-scale systems with many nodes and high data volumes. This is because the hashing function needs to be executed for every data item, which can be computationally expensive.
• Key collisions: Consistent hashing relies on a hashing function to assign data to nodes. If the hashing function is poorly designed or the data is distributed in a non-uniform way, it is possible for key collisions to occur, which can lead to data imbalances and reduced performance.

Ref: https://www.quora.com/What-are-the-advantages-and-disadvantages-of-consistent-hashing