search

dominictarr / scuttlebutt

peer-to-peer replicatable data structure
Other
1.32k stars 66 forks source link

Sharding #50

Open devrim opened 6 years ago

devrim commented 6 years ago

Hi Dominic, first of all thank you for the amazing library, which we successfully could integrate to our project, using streams on webrtc.

We are trying to implement a system where every browser becomes a node, eg. 1000 browsers, every 10 of them become a cluster, every peer in the cluster holds the same data (1/100th of the total), thus 100 clusters contain the full dataset. This is a network of browsers, therefore, each of them can appear/disappear at random (this example assumes not all 10 at the same cluster would disappear at once). We want to make sure when a browser joins the network, it holds only 100th of the dataset (numbers are figurative) so that they don't have to download/keep a lot of data. Peers in the cluster replicate, clusters will do the sharding.

Maybe a consistent hashring is the way to go, maybe finger table, I was wondering if you have given this a thought, if so, what would you advise?

dominictarr commented 6 years ago

yes, that is how i would do it. read the dynamo paper, it describes that architecture in detail - it's also essentially how bit torrent works from a big picture perspective