In order to ensure that user data in distributed mode will not be lost, we need a mechanism to ensure that CeresDB will not have multiple leader shard under any circumstances. etcd lock has been added in https://github.com/CeresDB/ceresdb/pull/706 , we refactor scheduler to implement failover based etcd distributed lock.
What changes are included in this PR?
Add shard watcher to get notify when shard lock is expired.
Refactor scheduler module, add shard lock delete callback to implement auto failover.
Which issue does this PR close?
Closes #
Rationale for this change
In order to ensure that user data in distributed mode will not be lost, we need a mechanism to ensure that CeresDB will not have multiple leader shard under any circumstances.
etcdlock has been added in https://github.com/CeresDB/ceresdb/pull/706 , we refactor scheduler to implement failover basedetcddistributed lock.What changes are included in this PR?
Are there any user-facing changes?
None.
How does this change test
Pass all unit tests and integration tests.