search

beeware / batavia

A JavaScript implementation of the Python virtual machine.
http://pybee.org/batavia
Other
1.39k stars 425 forks source link

Improve confidence in tests by enforcing that exception inside the tests are caught #792

Closed martica closed 5 years ago

martica commented 5 years ago

It is possible to have a transpilation test pass if line in the test throws an exception. As long as those exceptions match between JS and Python, that is considered a success. The behaviour will mask issues in tests and turn a test with a error in its expression into a passing test.

a = 1 + "2"
print(a)
print("Done.")

This will pass, because both JS and Python will throw a type error about the first line and they will match.

We should catch this exception and fail the test (and fix the tests that currently violate it).

freakboy3742 commented 5 years ago

Oh yeah - this is a big one. We're very much relying on "well written tests" to catch this edge case, which is a definite footgun.

The variant on this theme is "a test function that outputs nothing" - even if the behavior is completely different, the test will pass because there's no output, and no output == no output on both sides.

martica commented 5 years ago

I'm most of the way through correcting tests that don't catch their exceptions. Most are okay after adding the try, but there are some that have been hiding broken behaviour. A bunch in the tests of the time module. It wasn't importing time in most of the tests and throwing matching exceptions about that. :/