Leader entry created during election has a finite TTL and needs to be refreshed periodically.
Refresh operation should occur with an suitable margin before entry TTL elapses, to afford a couple retry attempts in case of temporary connectivity error.
An interesting case to consider is a split brain scenario that affects Akka cluster but not etcd cluster. In such situation two or more Akka cluster masters would contend to update the leader entry in etcd, which can be detected by using appropriate compareAndSet assertion: when a leader node notices that contents of the leader entry changed into a "foreign" address during the refresh cycle, it can tell that something fishy is going on. The "disposessed leader" should continue attempts to reclaim the entry. When the operator eventually kills of the extra clusters, leader of the surviving cluster will be able to restore the disvoery entries to correct state.
Leader entry created during election has a finite TTL and needs to be refreshed periodically.
Refresh operation should occur with an suitable margin before entry TTL elapses, to afford a couple retry attempts in case of temporary connectivity error.
An interesting case to consider is a split brain scenario that affects Akka cluster but not etcd cluster. In such situation two or more Akka cluster masters would contend to update the leader entry in etcd, which can be detected by using appropriate
compareAndSetassertion: when a leader node notices that contents of the leader entry changed into a "foreign" address during the refresh cycle, it can tell that something fishy is going on. The "disposessed leader" should continue attempts to reclaim the entry. When the operator eventually kills of the extra clusters, leader of the surviving cluster will be able to restore the disvoery entries to correct state.