apalache-mc / apalache

APALACHE: symbolic model checker for TLA+ and Quint
https://apalache-mc.org/
Apache License 2.0
443 stars 40 forks source link

Refactor and improve tests for the builder library #2580

Open shonfeder opened 1 year ago

shonfeder commented 1 year ago

The current tests use a lot of machinery around ScalaCheck that doesn't add any value afaict. On the contrary, it seem to make it considerably more difficult to read and understand the tests than it would be if we were using the usual PBT idioms advised by scalacheck. It is also difficult to know what has gone wrong when the tests fail, since the tests are doing too many things in each case and have undescriptive names.

Here is an example of a test case in the current approach:

https://github.com/informalsystems/apalache/blob/325f21c529307a0b9caa3d2311272ec9271d7a18/tlair/src/test/scala/at/forsyte/apalache/tla/typecomp/TestBaseBuilder.scala#L109-L159

Here are two cases in the approach I propose as an improvement:

https://github.com/informalsystems/apalache/blob/de453fe72282cab11ab3bfc78897ce16c75c0792/tlair/src/test/scala/at/forsyte/apalache/tla/typecomp/TestBaseBuilder.scala#L126-L154

Benefits of my proposed approach:

Kukovec commented 1 year ago

I'd like to see others weigh in on this too, but the tests you have proposed are not equivalent.

Ultimately, by my assessment:

Less than half the length (and that's after the addition of clarifying comments and imports that wouldn't be included in the actual code)

Yes, because it skips structure checking, and because it collapses all failure scenarios into one, which I'm not sure you can even do for all operators.

Better coverage, since we test for arbitrary invalid types

Untrue, since the mkIllTyped tests are just a decomposition of (nameType != elemType) (for the example above), and Generators.singleTypeGen generates arbitrary types.

More readable expression of the property being tested for Clearer naming of the test cases, so it is clear what test case or property is failing

Subjective, but I'm not opposed to more verbose test names, or splintering away IllegalArgumentException tests.