pablocostass commented 3 years ago

This PR extends PropEr with parallel execution. It is based on the work I did for my undergraduate thesis and has some limitations at the moment (e.g., targeted properties). Nonetheless, it should pretty much work (famous last words) and one can find the relevant options to test with parallel PropEr in the documentation of this branch, so please do take a look at it too.

The PR has two extra commits that I considered relevant but might be dropped upon request, those are:

Add verbosity to make test output (df7a7d8) Adds verbosity when running the tests, as that way it is easier to check at what point parallel PropEr failed in the suite.
Add a workflow to test with parallel PropEr (4c81af9) This commit adds a new workflow to be ran with Github Actions that uses 2 workers to run the test suite and ignores failures in the workflow; still, the warnings/failures are recorded by GA, so they will not be lost. This way, no PR would be blocked by some flakyness coming from parallel PropEr.

In short, this PR brings four new options to the table, all related to this new kind of execution:

{numworkers, <Non_negative_number>} sets the number of workers to use during testing.
pure (side effect free) or impure (with side effects) tells PropEr the kind of property it is going to test; impure properties start nodes to isolate the workers from the others and avoid possible test clashes.
{strategy_fun, <Strategy_function>} overrides the default function that divides the workload among the total of workers.
{stop_nodes, true | false} tells PropEr whether it should stop the nodes (if any was started) after finishing testing. This one is mainly used for proper:module/1,2, but it will probably be useful for some (e.g., build tools).

codecov-io commented 3 years ago

Codecov Report

Merging #263 (6bf55fe) into master (275c218) will increase coverage by 0.30%. The diff coverage is 94.02%.

@@            Coverage Diff             @@
##           master     #263      +/-   ##
==========================================
+ Coverage   88.65%   88.95%   +0.30%     
==========================================
  Files          14       14              
  Lines        4408     4601     +193     
==========================================
+ Hits         3908     4093     +185     
- Misses        500      508       +8

Impacted Files	Coverage Δ
src/proper.erl	`88.92% <94.02%> (+1.58%)`	:arrow_up:
src/proper_statem.erl	`94.67% <0.00%> (-0.39%)`	:arrow_down:
src/proper_typeserver.erl	`79.90% <0.00%> (+0.11%)`	:arrow_up:
src/proper_types.erl	`95.09% <0.00%> (+0.65%)`	:arrow_up:
src/proper_arith.erl	`92.70% <0.00%> (+1.04%)`	:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 275c218...6bf55fe. Read the comment docs.

pablocostass commented 3 years ago

@kostis I had forgotten to add the new make target to the Github Action workflow, so I did that after rebasing on top of the latest changes on the main branch.

I see that the Github Action is still failing due to the cache, however this time it is because the current cache is from the main branch and in said branch we do not build the PLT file with tools too, so I guess that in 7 days from now on we will have to trigger another run :/

kostis commented 3 years ago

I've pushed a small change to the Makefile that subsumes the addition of tools to the PLT -- so please rebase.

In order to achieve some progress, perhaps it's a good idea to temporarily disable the -Wunknown from the dialyzer call so that testing can continue with the tests.

pablocostass commented 3 years ago

Yup, that commit of yours helped @kostis, thanks! I rebased and pushed with the addition of a commit that disables the unknown warning, as you suggested. We can drop it in the future whenever we want :)

codecov-commenter commented 3 years ago

Codecov Report

Merging #263 (a2497b2) into master (e3a12c4) will decrease coverage by 3.30%. The diff coverage is 13.22%.

@@            Coverage Diff             @@
##           master     #263      +/-   ##
==========================================
- Coverage   88.47%   85.17%   -3.31%     
==========================================
  Files          14       14              
  Lines        4408     4586     +178     
==========================================
+ Hits         3900     3906       +6     
- Misses        508      680     +172

Impacted Files	Coverage Δ
src/proper.erl	`70.38% <13.22%> (-17.27%)`	:arrow_down:
src/proper_statem.erl	`92.77% <0.00%> (-1.91%)`	:arrow_down:
src/proper_gen.erl	`86.69% <0.00%> (-0.50%)`	:arrow_down:
src/proper_typeserver.erl	`78.76% <0.00%> (-0.46%)`	:arrow_down:
src/proper_erlang_abstract_code.erl	`94.24% <0.00%> (+0.25%)`	:arrow_up:
src/proper_gen_next.erl	`77.41% <0.00%> (+0.26%)`	:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update e3a12c4...a2497b2. Read the comment docs.

pablocostass commented 3 years ago

@kostis As the commit that had added the get_all_application_env() function also fixed some typos, I rebased to change it to only bring in the latter and also pushed some commits to address your comments.

As I am not really happy with how I did the refactoring of the common code that parallel_perform/2 had, I left that as a different commit. Review it and tell me if you thing something else should be changed.

proper-testing / proper

Extend PropEr with parallel execution #263

Codecov Report

Codecov Report