v1

[nicola]

• [ ] turn it into streams (peer.createPeerStreem() and this.stream)
• [ ] make it transportation agnostic (one should be able to use it with webrtc-swarm)
• [ ] abstract away identity
• [ ] benchmark/performance testing

of course this would not be possible without @diasdavid help!

[hackergrrl]

+100. I'm really excited about having this module available!

Streams are great! I used them for peer connections in secure-gossip -- it works really well and makes the module highly pluggable.

I also have some regrets that you may/may not be interested in:

Regret #1: I made the mistake of baking in the key library for identity. It'd be awesome if one could plug in their own key functions (generate, sign, verify, encrypt/decrypt) too.

Regret #2: I should've made createWriteStream and createReadStream methods instead of one-off publish and .on('message', ...) functions/events. hyperlog does this well.

[nicola]

@noffle thank you for your support!

on regret 1, right now it is kind of built on top of PeerInfo, however there is a function that is customizable (peerToId), the idea is to move in that direction. Yes, it would be awesome!

on regret 2, that is exactly the direction where I want to head, but I need more understanding / help. (maybe we can schedule a call next week?)

[hackergrrl]

I can try to summarize my thinking on r#2:

var b = new CyclonPeer()

// gets a readable stream of live reads from all peers in the network
// (aka. your PeerSet?)
//
// you don't know who originally published this, but this could be part
// of msg's format (wrap payload in a header that tells us who wrote
// this msg and their signature, etc)
//
// you could also just make 'b' a readable stream
b.getReadStream()
  .on('data', function(msg) {
    console.log('someone told us', msg)
  })

// stream that you write to, which publishes to peers using cyclon
// gossip
//
// you could also just make 'b' a writeable stream
var ws = b.getWriteStream()
ws.write('hello world!')

// Q: what happens if either stream gets closed? probably this'd
// invalidate your peer object and consider you disconnected from the
// network?

using streams also means you could make a pretty handy gossip CLI:

# subscribe to topic 'foo'
$ cgossip sub foo
hey
what's up
another published msg
^C

$ echo "hello network" | cgossip pub foo

some other module would be responsible for finding the peers (maybe ipfs, maybe discovery-swarm)

[nicola]

I will give a deeper read in a couple of hours. Some notes here

Have you seen @mappum's bitcoin-protocol ?

He has a different strategy (EncodeStream() and DecodeStream), I find that really interesting - maybe too lower level in our case.

On your question, the way I implemented it right now is the following: Peer A stores the MultiAddrs of peers in its partialView. If the oldest peer (Peer B) to which Peer A needs to connect to do shuffling, Peer A can see if it has a stream and if the stream has been ended it could try to connect again, if connection fails the peer is removed (although I just created an issue to test this)

[hackergrrl]

Have you seen @mappum's bitcoin-protocol ?

He has a different strategy (EncodeStream() and DecodeStream), I find that really interesting - maybe too lower level in our case.

Cool! Matt's approach makes sense: each encoder/decoder is a transform stream that accepts data in one format and outputs it in another. Does that make sense here? As in, does a publish or subscribe operation perform encoding/decoding? Maybe I've misunderstood your point.

My thinking is that each peer instance has its own pair of streams: one for reading incoming messages from the gossip swarm, and one for writing outgoing messages to be shared with the swarm. Does that make more sense?

[nicola]

I wanted to get this working first to get this type of feedback, otherwise I would have done self-bikeshedding not concluding much, can't wait to go over iteration 2!

I haven't actually sat down and read hyperlog yet!

Question 1 - keeping everything in one layer vs separating layers @noffle, that absolutely make sense. The way I see this though is that this lib should just do membership management, propagating messages to peers is a layer above this. In other words, Cyclon provides each peer a partial view and keeps this partial view fresh by pinging old peers and asking for new peers (every x amount of time).So, I wonder if maybe we can use the same stream like you seem to suggest, or pipe this stream in the communication stream).

Question 2: internal stream? Should CyclonPeer writing to its stream to announce that it is starting shuffling, or that it is updating the age of its peers or should we only have streams for external communication purposes only? (I was thinking if we are thinking about replicating a peer - let's say for fault tolerance, than you could truly replicate a peer this way). It is a sort of stream of the internal log. (am I putting too many streams?)

Question 3: how many streams? Is the idea that we should have one pair of streams or one pair of streams for each.. connection?

in the following ⬇️ how will this stream get data into? (I have an idea)

b.getReadStream()
  .on('data', function(msg) {
    console.log('someone told us', msg)
  })

[hackergrrl]

The way I see this though is that this lib should just do membership management, propagating messages to peers is a layer above this (I wonder if maybe we can use the same stream like you seem to suggest, or pipe this stream in the communication stream).

Good thinking: I think this is a smart place to separate. You could expose the duplex streams of the current peers, and use them in the messaging layer.

in the following ⬇️ how will this stream get data into? (I have an idea)

b.getReadStream()
  .on('data', function(msg) {
    console.log('someone told us', msg)
  })

The underlying module implementation will implement a readable stream[1] which will inject the data to be made available to the reader using the module.

[1] https://nodejs.org/api/stream.html#stream_class_stream_readable_1

[daviddias]

Hey @nicola, this is looking really great, congratz on the great work and it is always a pleasure to help :)

I understand that you want to support WebRTC or just be generally transport agnostic, as you know libp2p is built exactly for that :) I'm looking into bringing some WebRTC transport for libp2p this week, first creating a full mesh and then another to enable packet switched over WebRTC data channels (continuing the work in webrtc-explorer), what kind of Peer channels are you looking for?

On identity management, libp2p can (the way that go-ipfs does) handle authentication for you and we want to bring this to JS, wanna help us build secio in JS? As for accounting, that you can handle in your app by picking the priv+pub keys from PeerInfo.

On benchmark/performance testing, we want to improve even more at that, get some PlanetLab like testing for our p2p protocols, is this something related to what your are thinking?