Open gavin-norman-sociomantic opened 5 years ago
Note that we don't need to periodically iterate over the set of the nodes, but we can use node's client's connection notifier to sync the orphaned records when the node reconnects with the lost node.
One thing to worry about here (and maybe it's not worth pursuing at this level) is the network partition problem: if you have two nodes A and B which both can talk only to a subset of clients, and which can't talk to each other, both will receive updates for their records and for the records to the node that's not available. Then on the network recovery, both node A and node B will have the new state for the B's records.
The redistribution system could probably be rephrased to work with this too: When the hash range of the node changes, it iterates over its channels, puts out-of-range records into the orphaned records store and lets magic do the rest.
Ah, yes, that's a nice consequence.
Resilience to node outage for Put requests is actually quite simple.
On the client side:
no_node
error or a connection error, the client picks another node (using some deterministic algorithm) and sends the record there. (Possibly repeating, if multiple nodes are out.)On the node side:
.nodes
file at startup, connecting to all other nodes.