Closed andriytk closed 5 years ago
merged
Okay. Looks good to me.
Lease timeout was increased because the io sometimes cannot complete within the timeout - yes. As a result, PAXOS cannot complete and the RC restarts. You could see a lot of such restarts in halon decision log and system log before the commit 8d71b602 here in the GitLab tests artefacts.
Okay, so earlier the lease timeout was increased because the halon keepalives were not processed in given time due to heavy io load is it?
assigned to @mandar.sawant
@mandar.sawant could you review it, please?
added 1 commit
added 1 commit
changed title from HALON-900: fix multi-nodes TS cluster bootstrap to HALON-900: fix multi-nodes TS cluster bootstrap{+ regression+}
added 1 commit
added 1 commit
changed the description
The same lease timeout which we increased in the commit 8d71b602 is also used by the ambassador to ping the replicas and determine the leader. So the leader determination increased as well along with the RC startup time. As a result, the satellites startup timed out during the bootstrap.
Now we just hard-code the initial ambassador timeout to 1 sec. It seems to be enough to quickly and reliably determine the leader during the bootstrap and also break the dependency of this determination process on the lease timeout configuration.