zpqrtbnk
commented
11 months ago Analysis: troubleshooting reveals a range of issues.
- (important) In some situations, failing to complete connection to the next cluster during a failover operation leaves the client hanging instead of flowing to yet another cluster
- (minor) Logging reports that the client switched to the
Startingstate when in fact the state did not change - (important) Cancellation token not properly propagated to heartbeat means that heartbeat can hang on a cluster even after we have stopped trying to work with that cluster
- (important) Connection not properly teared down if it is established yet the members view is not received
All these issues revealed by a new test that runs failover scenarios over mocked clusters (so we can simulate not sending the members view).
From Java's API-2171 if the client manages to initiate a connection to the next cluster during a failover operation, but the cluster fails to raise the members view event soon enough, the client may end up shutting down or worse, hanging, instead of nicely flowing to yet another cluster.