search

palantir / atlasdb

Transactional Distributed Database Layer
https://palantir.github.io/atlasdb/
Apache License 2.0
54 stars 10 forks source link

Implement a variant of Paxos that supports dynamic cluster reconfiguration #2473

Open kasper0406 opened 7 years ago

kasper0406 commented 7 years ago

The current implementation of Paxos assumes a static cluster configuration. This means that Paxos logs have to be manually truncated every time a new server is added (for example if it was reinstalled due to a crash, or if the number of Paxos nodes is increased).

I believe this to be true for both the time lock server and for an internal project.

It is dangerous to rely on manually remembering to truncate the Paxos logs when joining in new servers.

nziebart commented 7 years ago

We definitely don't support dynamic reconfiguration but curious what you mean by needing to truncate the logs when adding another server. Do you truncate the logs for all the servers?

tpetracca commented 7 years ago

You have to truncate when a pre-existing host loses its paxos logs. More specifically, you need to truncate any time a node has "forgotten" about a proposal/agreement it previously made.

jeremyk-91 commented 7 years ago

See #1495 for a bit of discussion regarding log truncation.

About joining in new servers: this can also apply if you're increasing the size of an odd cluster by 2+ or an even cluster by 3+, without a paxos round in between.

If you had a cluster of 2k+1 nodes and you bump it to 2k+3, then you could have k+1 of the nodes reach (old) quorum on a given sequence number, and the remaining k nodes plus the 2 new ones reach new quorum on the same sequence number.

hsaraogi commented 7 years ago

1.We can bundle the CLI with the internal large products timelock-server.

  1. We will not run into this issue in cloud deployments but we should bundle this CLI with the timelock server for on-prem deployments.

We can test this scenario via Jepsen tests and run the CLI to verify that this fixes it.

@carrino for SA.