python 3 and a question

[snapo]

Hi, i did just wanna ask if you at some point will upgrade it to python3?

Also i would like to ask about the algorythm about data integrity / rebalance / re-distribution: in the case (for simplicity 5 clients) where all clients have received a key with node.sendall("hello!", duplicates=3) so 3 clients have the "hello!" in there key value store. If client 3 (for what ever reason) dies, how does it rebalance it to 3 again? i dont see checks where this is happening (maybe i overlooked something). Also if a new client (6) joins, how does the network rebalance that to the new client?

Thanks for the answer already...

[Shaptic]

Hey Snapo,

I am planning on upgrading to python3, but I want to stabilize the current build. No ETA on that yet as this is a side-project for me. With regards to your question:

I think you may be misunderstanding the purpose of message sending. The purpose of the Cicada protocol is to provide routing. It's not a key-value store like a traditional DHT is. There is no API corresponding to a sendall operation for the high-level SwarmPeer object. There is:

• send([target], [message], [options]) — routes a message through the network to a single target. If you pass duplicates=n to this, the message will take n paths to reach its destination target. It's intended to raise routing security, as multiple paths == less opportunity to run into an adversarial peer.
• broadcast([message]) — this sends a message through the network to all nodes in an optimized way. The duplicates kwarg doesn't make sense here, since everyone will receive the message anyway.

If you did want the message to be stored across multiple nodes, that would have to happen at a higher level than what Cicada currently provides.

I hope that clears things up!

[snapo]

@Shaptic Thank you very much for your explanation, yes indeed i am looking for for a DHT with automated shard rebalancing. I did think first it wouldnt be that complicated but after the more i read the more i understand the problem on distributed data sources :-( .

Still compliment very great project you have there.

[Shaptic]

Thanks. :)

I see what you're talking about! That's definitely an interesting idea. In theory, here's what I would imagine:

• Underlying DHT, like Chord or Kademlia
• Notification system to the higher distribution layer, so that peers are notified when neighbors leave the swarm.
• When you put(data, number_of_copies), the data gets placed on the peer matching hash(data), and every following peer up to the number of copies (or some other deterministic duplication scheme)
• When the main peer goes down (the one corresponding to hash(data)), peer newly responsible for the data already has it. Further, everyone knows there's one less copy now, so it's added to another peer and "redistributed."

I'm sure this is something you've already considered, though! I'm curious to know what the findings of your research are. :) either way, good luck with your project!