josefs / Gradualizer

A Gradual type system for Erlang
MIT License
609 stars 35 forks source link

Use Common Test for testing when Gradualizer should pass, fail, and its known problems #567

Open erszcz opened 1 month ago

erszcz commented 1 month ago

This PR adds ability to test Gradualizer functionality with Common Test (aka CT), which reports results in a way that enables much more convenient comparisons between builds. CT generates HTML test run reports. The Makefile rule, make ct, labels test runs with git commit info and a modified annotation, if a run tested a modified working directory. All in all, when introducing a change, this testing framework makes it easier to spot what broke and compare results between various builds.

For now, it's not integrated with CI.

Usage:

make ct

or if we're only interested in a specific suite:

rebar3 ct --suite=known_problems_should_pass_SUITE --label="git: $(git describe --tags --always) $(git diff --no-ext-diff --quiet --exit-code || echo '(modified)')"

Example results

Command line:

Screenshot 2024-06-04 at 09 32 35

Please note that line numbers are always the same, since the CT tests (test functions) are generated from a template. However, the suite and test name are sufficient to identify the typechecked file the CT test corresponds to.

HTML listing of all runs (just 2 in this case):

Screenshot 2024-06-04 at 09 41 44

Two runs compared side by side:

Screenshot 2024-06-04 at 09 35 28 (2)