Benchmarks

[defuse]

Benchmark all of the implementations and produce a report.

Also include other, existing, implementations of scrypt, like the following, which may lead to performance improvements:

https://github.com/dchest/tweetnacl-js/blob/master/nacl-fast.js#L58 https://github.com/dchest/scrypt-async-js/blob/master/scrypt-async.js#L203

[defuse]

These benchmarks will be important for our recommendations on whether or not to actually use the code. If too slow, releasing the code without strong warnings against use could be dangerous.

[defuse]

I decided on the following design for benchmarking.

Each implementation's command-line interface supports a 'benchmark' command, which takes the number of iterations to time, as well as the operation (yescrypt w/ parameters, pwxform, or salsa20_8). It times how long it takes to perform that operation and then returns the number of calls per second as output.

This design ensures that each implementation can do timing and loop-unrolling in the way that's best for that's best for that implementation. Implementations are also responsible for not producing incorrect output, e.g. if the iteration count is too small, it should output an error message instead of the inaccurate result.

There will be a "collect benchmarks" script that runs a bunch of benchmark commands, calculating averages etc. for each implemenation and function pair, using pre-defined yescrypt parameters, and then generating an ASCII report.

[defuse]

The SIMD implementation is a little tricky to benchmark, since it has to be done inside firefox, and there's no CLI interface to running a JS script. I'll most likely do it by creating a static page that produces a result's JSON thing that you have to paste into a text file for the "collect benchmarks" script to read.

(In the future we could do something like spawn a web server and have the page make a POST request with the results, and after we get it, kill firefox).