search

heapwolf / level-replicator

WIP; Eventually consistent log-based multi-master replication for levelDB (@Level)
MIT License
70 stars 11 forks source link

what happens if connection fails before replication completes? #1

Closed dominictarr closed 10 years ago

dominictarr commented 10 years ago

From your readme I gather that this scans the database in reverse to find the most recent record that the other end already has... are the incoming records written or buffered until the latest record is found?

heapwolf commented 10 years ago

Replication starts from oldest to newest. The replicating client does a read stream of { reverse: true, limit: 1 } from the remote, when that record is found, the next applicable group of operations are batch executed.

Unfinished replications are expected.

On Monday, April 21, 2014, Dominic Tarr notifications@github.com wrote:

From your readme I gather that this scans the database in reverse to find the most recent record that the other end already has... are the incoming records written or buffered until the latest record is found?

Reply to this email directly or view it on GitHubhttps://github.com/hij1nx/level-replicator/issues/1 .

Paolo Fragomeni Founder, Here is How http://hereishow.to

github.com/hij1nx twitter.com/hij1nx

dominictarr commented 10 years ago

so, you request the latest record from the server, and then send everything that you have got that came after that.

dominictarr commented 10 years ago

what if there are more than one client, and you do not have the latest record on the sever, because it came from another client?

heapwolf commented 10 years ago

Statistically it will be possible that a master connects to another master and doesn't have any news to share. In which case it should simply disconnect.

This is a non-debilitating compounding inefficiency based on the size of the network. It can be optimized in lots of ways. For instance, the connection frequency could be optimized using the assumption that the number of local writes should be proportionate to the number of remote writes, but i haven't written any good tests for any of my optimization theories.

heapwolf commented 10 years ago
  1. PEER-1 creates A, B and C.
  2. PEER-2 collects A, B from PEER-1 and disconnects, modifies B.
  3. PEER-3 collects A, B and C from PEER-1.

PEER-3 is late to the party, but when it connects to PEER-2 it discovers it needs to update B. PEER-1 eventually connects to PEER-3 and updates. PEER-3 connects to PEER-1 and has nothing to do, it disconnects.

So, each peer is continuously pulling from some random peer. This model gracefully handles partitioning; great for scenarios where eventual consistency is acceptable, but obviously not for scenarios where strong consistency is a requirement. I've been working on that separately.

dominictarr commented 10 years ago

I've been thinking a lot about how to test replication stuff and databases in a realistic way. I think the answer has to be to conduct a simulation of the database usage... You have a bunch of actors, who use the database in a given pattern. (write with a given prob, read with a given prob, replicate with a given prob)

That will verify things are working and be good for benchmarking, although it will be hard to debug something that isn't deterministic...

I think strong consistency is overrated :) It's only viable within datacenters, and the most interesting distributed systems are also decentralized systems.

anyway, the hard part about testing is finding edge cases / race conditions, and verifying that it all works... normally I've done tests where you replicate, check it's eventually consistent, replicate, modify, replicate again replicate modify, replicate with another node that has new data, then replicate again with the first and then check it's right... It's a pain in the butt writing that many tests, but I've always found bugs with something until I've gone that far...

heapwolf commented 10 years ago

A series of tests that asserted the basic requirements of any EC solution would be 10x more valuable than a single effective EC solution :)

I agree that it is overrated, but strong consistency isn't really something you can write off entirely. For instance: financial transactions where you need atomic increments or decrements.

Im pretty sure you are calling for a canonical test suite, which I think would be completely awesome.

dominictarr commented 10 years ago

yeah... working on stuff like this right now, well, for benchmarking databases. should have something by decentralize camp.

jasonm commented 10 years ago

Jepsen looks helpful for simulating partitions:

https://github.com/aphyr/jepsen http://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-and-the-perils-of-network-partitions http://aphyr.com/tags/jepsen

heapwolf commented 10 years ago

If #4 & #5 go as planned, ldbcmp should tell is if this works.