clearmatics / zeth

Integration of Zerocash on Ethereum: https://arxiv.org/abs/1904.00905
https://clearmatics.github.io/zeth/
GNU Lesser General Public License v3.0
56 stars 26 forks source link

Minimal benchmarking framework #230

Open AntoineRondelet opened 4 years ago

AntoineRondelet commented 4 years ago

That would be ideal to have some minimal benchmarking framework. As of today, much benchmarks were carried out manually, but having a better/automated way to run benchmarks would make our life easier and would allow us to better capture the impact of the various PRs on the overall performances of the project.

I'd consider this a first good issue to tackle for anyone willing to get involved in Zeth. A good/dummy first iteration would be to have some timers added to the cpp code if compiled with a BENCHMARK option. That would time remarkable operations for the given set of protocol parameters. The result of this benchmark could then be written in a benchmark folder containing, say, a csv file where each line would correspond to a PR. That way we'd have some basic tracking of the impact of each PR on the perfs of the code, and ultimately track perfs gains/loses across releases. Having this data recorded in a csv would pave the way for basic graphs with python scripts to make the results digestible and easy to share.

This obviously describes some very basic first step for the benchmarking framework, something more elaborate would need to be done later down the line. @dtebbs any thoughts, or things to add?

AntoineRondelet commented 3 years ago

Look into:

Ideally, we'd like to benchmark things like:

Doing such benchmarks would allow to better track perfs variations across releases and even track down perfs associated to each PR (if the benchmarking tool is incorporated to the CI and ran on-pull-request)