IPFS performance improvments and experiments

[musalbas]

We need to get to the bottom of why "providing" time for the experiments (https://github.com/lazyledger/ipld-plugin-experiments/pull/9) is different to the MVP, and in general, work towards more confidence in the performance and scalability of IPFS.

Experiments that can be done:

• Remove sleep from ipld-plugin-experiments https://github.com/lazyledger/ipld-plugin-experiments/blob/a4c8edf11ac9a465b0053f124804a64c637acb0e/experiments/clients/main.go#L62
• https://github.com/lazyledger/ipld-plugin-experiments/issues/4
• https://github.com/lazyledger/ipld-plugin-experiments/issues/10 (use NMT in experiments)
• In lazyledger-core MVP, what's the total time to sample a chunk, when the sample request is made immediately after the block is generated (but before provided is finished) and when the block interval is 30 seconds?

Experimental improvements:

• What's the providing time when a private IPFS network is used?
• What's the providing time when Hydra nodes are added as bootstrap nodes?
• What's the minimum providing time when multiple validators are providing the same block?
• What's the providing time when new DHT (crawler-based) client used?

[Wondertan]

In lazyledger-core MVP, what's the total time to sample a chunk, when the sample request is made immediately after the block is generated (but before provided is finished) and when the block interval is 30 seconds?

@musalbas, I guess you mean unconnected node. I've done enough experiments to say that it would take >1.5min

[Wondertan]

But that's on real IPFS network, in the case of a private network the time would be much lesser, but we still need to do deploy it and exec experiments

[Wondertan]

If we imagine that our network grows to the size of the IPFS's one, I am confident that there is no way around it, besides moving on new DHT client with full routing table

[musalbas]

We concluded the following: DHT providing was still slow only for the first request in the DigitalOcean experiments, but it took ~3s in subsequent requests because BitSwap kicked in, to bypass DHT.