IPNS Benchmarking

[aschmahmann]

There is on-going work to make IPNS over PubSub have a faster initial resolution time by adding persistence to PubSub (e.g. libp2p/go-libp2p-pubsub#171). Additionally there is work on the libp2p testlab (https://github.com/libp2p/testlab) to enable benchmarking of libp2p based systems. It would be fantastic if we could utilize the testlab to benchmark the performance of IPNS over DHT vs IPNS over PubSub (current) vs IPNS over PubSub (new) to understand our actual improvements and what aspects of the system are still holding us back. This can also serve as a good exercise case for the testlab.

Audience: IPNS and testlab devlopers

Impact: Enables IPNS and testlab developers to better target areas for improvement. These results spill over into creating more metrics driven testing of libp2p-based components generally and turning our focus to the particular systems that are on our critical paths.

Stakeholders: go-ipfs team, testlab/libp2p team

[jimpick]

Related, js-ipfs has a Q2 OKR get the new benchmark suite to run via continuous integration... (@alanshaw)

https://docs.google.com/spreadsheets/d/1YSeyWqXh3ImanRrTkYQHHkCofiORn68bYqM_KTLBlsA/edit#gid=274358435&range=B15

https://github.com/ipfs/benchmarks

Potentially there's some overlap?

[momack2]

Given that testlab is up and running and many of these pieces have at least working forks / proofs of concept - would be awesome to push this thread forward!

[aschmahmann]

@momack2 @jimpick that would be great. I even have a go-ipfs fork that supports testing IPNS over DHT and the new IPNS over PubSub at https://github.com/aschmahmann/go-ipfs/releases.

[daviddias]

It should be simple to add benchmarks to -- https://github.com/ipfs/benchmarks --. You also might want to add IPNS tests to -- https://github.com/ipfs/interop -- and I would love to have a -- https://github.com/ipfs/integration -- for tests that test things on the live network.

[aschmahmann]

@daviddias I think there are already IPNS interop tests for the DHT (and possibly for the older version of PubSub). We're just now getting the faster IPNS over PubSub off the ground in Go, hopefully we'll be able to tackle JS next quarter and definitely add those to the interop tests.

I'm not familiar with https://github.com/ipfs/benchmarks, but my understanding of one of the testlab's goals is to be able to specify network configurations with which to run our various tests. A longer term goal here is to be able to understand how our implementations fare for large and small networks, partitioned/NATed networks, etc. so that we have more meaningful descriptions of what is meant by "the DHT is slow".

@jimpick I think you've spent a bunch of time looking at the testlab. Does this seem doable in a relatively short time span?

[jimpick]

I haven't spent enough "quality time" with Testlab as I intended to this quarter. I do want to get going with it next week however. Past experience building this type of testing infrastructure tells me that initially it will be slow going, but the initial investment will pay off in the end. I'm taking 1 week off before team week, but I'm hoping to at least be up-to-speed enough with Testlab so we can do some hacking on it during team week.