Either every day or with every new commit in one of IPFS' exchange interfaces repos (mainly Bitswap and Graphsync) or IPFS itself, run a test case to evaluate their performance.
Publish a comparison of the results (Bitswap v.s. TCP v.s. Graphsync) in a public website for anyone to be able to track their current performance, and improvements with every commit.
We have to agree on the test cases to run for the benchmark. But a good start would be:
1:1 exchange with 100ms for B=50Mbps/100Mbps and three well known datasets up to 1GB from https://awesome.ipfs.io/datasets/ (so anyone is able to replicate the experiment).
An additional experiment n:1 in Bitswap to evaluate whether it is leveraging multi-path exchanges and the benefit of this.
We may be able to leverage Testground as a Service to trigger the test with every new commit.
BONUS: If we can add to this benchmark an experiment that perform an exchange between two nodes at different locations through the real IPFS network (instead of using Testground), we would be able to evaluate how the status of the network may affect the file exchange (although this may not be worth the effort considering the added value it really brings)
With #32, #32, #33, the testbed is getting into great shape, and considering @hannahhoward and @acruikshank's amazing job visualizing Graphsync's performance compared to TCP for several files and network conditions, it'd be great to have a public IPFS file-sharing performance comparison dashboard similar to this one for NodeJS: https://benchmarking.nodejs.org/
My proposal would be:
We have to agree on the test cases to run for the benchmark. But a good start would be:
We may be able to leverage Testground as a Service to trigger the test with every new commit.
BONUS: If we can add to this benchmark an experiment that perform an exchange between two nodes at different locations through the real IPFS network (instead of using Testground), we would be able to evaluate how the status of the network may affect the file exchange (although this may not be worth the effort considering the added value it really brings)