[3.2] benchmarking framework and its use in benchmarking crypto primitives

[linh2931]

Resolve https://github.com/eosnetworkfoundation/mandel/issues/786 and partially resolve https://github.com/eosnetworkfoundation/mandel/issues/780.

Implement a benchmarking framework to help writing benchmarking code by reusing common measuring functions and unifying reports. After the PR is approved, the framework can be used to benchmark other functions.

For end users

Below are a few examples

Help

./benchmark -h
benchmark command line options:
  -m [ --module ] arg       the module to be benchmarked; if this option is not
                            present, all modules are benchmarked.
  -l [ --list ]             list of supported modules
  -r [ --runs ] arg (=1000) the number of runs per function
  -h [ --help ]             execute the benchmarked functions for the specified
                            number of times, and report average, minimum, and
                            maximum time in nanoseconds

Benchmarking All Modules (functions)

./benchmark
function                      runs               avg              min              max

alt_bn_128:
alt_bn128_add                 1,000         6,467 ns         3,727 ns       113,625 ns
alt_bn128_mul                 1,000       251,105 ns       235,337 ns       395,856 ns
alt_bn128_pair (1 pair)       1,000     6,405,913 ns     6,354,466 ns     8,265,015 ns
alt_bn128_pair (10 pairs)     1,000    25,379,638 ns    25,236,388 ns    36,013,377 ns

modexp:
1024-bit width                1,000       389,252 ns       388,298 ns       396,263 ns
2048-bit width                1,000     2,914,494 ns     2,910,697 ns     2,999,909 ns

Benchmarking Particular Module with Specific Number of Runs

/benchmark -m modexp -r 500
function                      runs               avg              min              max

modexp:
1024-bit width                500         441,057 ns       388,091 ns       818,064 ns
2048-bit width                500       2,914,402 ns     2,911,318 ns     3,001,094 ns

For developers

Please see alt_bn_128.cpp and modexp.cppfor examples.

The general approach to adding code to benchmark a set of functions is

1. Implement a function where for each function to be benchmarked, prepare input, define a lamda to call the function, and call common benchmarking.
2. Add an entry to modules in benchmark.cpp.

[heifner]

Should we consider using https://github.com/google/benchmark which has built in support for preventing the compiler from optimizing out what you are trying to benchmark as well as other useful features for micro-benchmarking.

[linh2931]

The Google microbenchmarking library looks great. It has lots of features and more code. Just wondering if it is brought in, would it increase dependencies and build time?

[stephenpdeos]

We will not move forward with Google benchmarking framework at this time, because it doesn't seem to have enough value add compared to current code used.

[spoonincode]

I wonder if there should be a machine-readable output (like JSON) to make it easier to ingest in to some reporting framework.

[linh2931]

I will add a -j option for JSON format output.

[larryk85]

I wonder if there should be a machine-readable output (like JSON) to make it easier to ingest in to some reporting framework.

That would be nice to have, but maybe we can list that as a phase 2 item for after this is merged and done.

[linh2931]

Close this as it is being ported over to Leap as https://github.com/AntelopeIO/leap/pull/17.