This PR adds ability to test Gradualizer functionality with Common Test (aka CT), which reports results in a way that enables much more convenient comparisons between builds. CT generates HTML test run reports. The Makefile rule, make ct, labels test runs with git commit info and a modified annotation, if a run tested a modified working directory. All in all, when introducing a change, this testing framework makes it easier to spot what broke and compare results between various builds.
Please note that line numbers are always the same, since the CT tests (test functions) are generated from a template. However, the suite and test name are sufficient to identify the typechecked file the CT test corresponds to.
This PR adds ability to test Gradualizer functionality with Common Test (aka CT), which reports results in a way that enables much more convenient comparisons between builds. CT generates HTML test run reports. The Makefile rule,
make ct
, labels test runs with git commit info and a modified annotation, if a run tested a modified working directory. All in all, when introducing a change, this testing framework makes it easier to spot what broke and compare results between various builds.For now, it's not integrated with CI.
Usage:
or if we're only interested in a specific suite:
Example results
Command line:
Please note that line numbers are always the same, since the CT tests (test functions) are generated from a template. However, the suite and test name are sufficient to identify the typechecked file the CT test corresponds to.
HTML listing of all runs (just 2 in this case):
Two runs compared side by side: