How to integrate performance tests?

[magicDGS]

For some kind of statistics, we are planing to perform a cache-based solution to improve performance because the operations are time consuming (e.g., #15). Nevertheless, we want to be sure that the performance is really better with the cached solution, and that there is no regression if bug-fixes or other changes are applied to the code. Thus, we need to come up with an idea to test performance. My main idea follows this schema:

• Use JMH for implementing benchmarks inside the unit test framework.
• Within the unit test, compare the measurement (e.g., average time) between the cached and no-cached version. Fail if the performance is not better.
• TestNG test suite for running performance tests alone. This will be useful for separate the tests on Travis and check for regression, and to do not include this tests into the coverage report.
• Use gradle and JMH create a runnable benchmark.jar to generate reports and upload to the documentation (if needed).

One interesting idea is to separate benchmarks from tests as suggested here. But we require to test if there is some regression.

Another idea is to convert popgenlib into a multi-module project with the following sub-modules: popgenlib-core and and popgenlib-benchmark. The core will contain the base code, and the benchmark JMH code to package a benchmark.jar and to run performance-unit-tests. This structure will help also to separate the cached versions in another module if needed for minimize the burden of including them in downstream projects without that requirement.

[magicDGS]

Any thoughts on this @JPinzon01? If it helps, you can see my developments of one of the cached classes and the tests in this branch: https://github.com/magicDGS/popgenlib/tree/dgs_wattersons_theta_cached