Primitive Benchmark

[isheff]

Description

We should establish some way to test how "good" (throughput? latency?) Typhon is (relative to Tendermint). This means creating some kind of benchmark now that can at least measure tendermint, and then as we build prototypes, we should be able to see how well they compare.

Some features we might want include:

• ability to test gossip/storage vs. consensus/ordering layers separately
• ability to meaningfully calculate performance for many machines, ideally without actually using many machines.
• ability to abstract over optimizations that Tendermint's software has made, and we can also make, but haven't yet done in our prototype (e.g. fast marshaling / unmarshaling of messages).

Sub-tasks (not in particular order)

• find out whether there is some benchmarking tool implemented by others (e.g., Tendermint themselves) that we can use
• determine suitable performance metrics that will allow comparison
• determine a concrete set of behaviors that we want to benchmark (filter out unnecessary complexities)
• determine a set of functional/non-functional requirements for the benchmarking tool
• design the benchmarking tool
• implement the benchmarking tool
• perform a comparison study

Definition of Done

We have a document with a concise and clear comparison study between Tendermint (what we use for Namada) and Typhon (what we use for Anoma).

[FatemeShirazi]

https://arxiv.org/pdf/2105.11827.pdf have done some measurements, so that would be a good place to start for getting ideas about performance metrics, behaviours, and requirements.

[FatemeShirazi]

If there is working hackmd document, could you please add a link here?

[nzarin]

A not-updated version here : https://hackmd.io/Rr60ib5_S3qvvdWmAC_GYQ