Old nodes are kept forever and never removed from the cluster view. As the number of node increases, the number of async thread required to perform the health check increases and flooded the log file with rejected exceptions (thread pool exhaustion).
Cluster validation should be clever and not use all the threads to perform internal validation of other peers. Pacing, throttling could be use to be scalable.
Old nodes are kept forever and never removed from the cluster view. As the number of node increases, the number of async thread required to perform the health check increases and flooded the log file with rejected exceptions (thread pool exhaustion).
Cluster validation should be clever and not use all the threads to perform internal validation of other peers. Pacing, throttling could be use to be scalable.