Design and build the harness for pubsub tests

[daviddias]

Notes from the call

• We need to figure out a way to spawn a lot of long-running nodes; hundreds, potentially THOUSANDS. Begin with your machine: "Hey, I want to spawn n nodes in my machine".
• These nodes will have to be non-IPFS nodes, just libp2p.
• You could use Kubernetes for this.
• We need this asap in order to test our design decisions
• Deploying and clustering can be separated out. You need a way to control the cluster, in addition to spawning and deploying nodes.

[gavinmcdermott]

Thanks for posting—I also recalled something about:

• Bitswap having a script to spin up nodes
• go-ipfs having something for this purpose (was it IPTB?)

Any clarifications or links to relevant repos would helpful

Since the harness is a must have for people to implement and test scenarios, and for my own (selfish) purposes of diving into libp2p and getting myself familiar with the ecosystem, I'd be happy to take a run at building the initial test harness to keep things moving forward (though there may be a couple questions along the way).

@diasdavid : I thought it was mentioned that you might take a crack at this, but it also sounds like there is a lot going on. So if this isn't duplicating efforts or working on something that you intended to do, I'd be happy to get this going. Let me know!

[daviddias]

IPTB exists for go-ipfs, but here we would be shooting for a straight network of libp2p nodes. Here is the repo https://github.com/whyrusleeping/iptb

I've created this script for bitswap -- https://github.com/ipfs/js-ipfs-bitswap/blob/master/test/utils.js#L96 -- that can be repurposed, because it is essentially spawning libp2p nodes, JavaScript ones though :)

@gavinmcdermott thank you for volunteering, that would be excellent! I'm happy to review and provide help with support, as human documentation or with code at any time.

[Kubuxu]

As you want to spawn much more nodes that we usually do in go-ipfs, you would have to go for something lighter, maybe clustered approach:

Multiple nodes (few hundreds) inside one process (node or go) and then some of them can connect to the outside, with other clusters like that.

IPTB creates full go-ipfs nodes and has few options for separating them: none or docker. For a long time I am planning to implement network namespaces separation option but I haven't time do do it, it would have benefits of docker separation with no additional overhead.

It is also quite important to simulate real world quite accurately if you want to use it for testing, this means latency, packet drop and nodes going up and down, this is something that we missed during work on cjdns (which is written in C and we were able to create networks of millions of nodes during tests) and it turned out that our path searching implementation has some problems with that. Only after about 2k nodes joined real world network we started seeing those problems.

[gavinmcdermott]

Thanks for sharing @Kubuxu, much appreciated.

[jbenet]

I would also look at kubernetes, that has all the clustering that we may want to do, I think.

cc @hermanjunge as he may be interested in this too. He is doing this for ipfs-cluster On Tue, Aug 16, 2016 at 21:08 Gavin McDermott notifications@github.com wrote:

Thanks for sharing @Kubuxu https://github.com/Kubuxu, much appreciated.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/libp2p/pubsub/issues/10#issuecomment-240286158, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIcoQIWAmU6QF6TeTglRF0bRFfhvzA9ks5qgl8kgaJpZM4JlJbl .

[ghost]

How can I be of help?

[gavinmcdermott]

@hermanjunge, gracias. Feel free to email me so we can sync up and keep any unrelated spam off the thread.