Open dobesv opened 4 years ago
@rahuldesirazu I face the same situation yesterday, and got to this workaround:
This way this new master will join others (normally as NOT_PARTICIPANT), and this will allow raft consensus to complete. From there on, you can now remove this 'incompleter' using yb-admin ... change_master_config REMOVE_SERVER $bad-master$, and proceed to actually initialize it right (ie. starting it in shell mode, etc)
Best regards Pablo
I accidentally put our cluster into a state where a defunct master cannot be removed, and new masters cannot be added. Although this is probably a niche case it would be nice to know if there is a workaround.
The issue arises with these steps (in theory):
At this point, it refuses to allow adding or removing masters from the configuration, giving the error shown in the transcript below.
I wonder if some kind of special case is needed for the raft logic during a
REMOVE_SERVER
operation such that it does not require the participation of the server being removed?Perhaps a way to temporarily reduce quorum requirements for the cluster or force reset the whole master config to contain only the current leader would allow me to escape this situation.