Small samples - Githubissues

15r10nk commented 1 year ago

generate small samples to improve test coverage.

The samples where generated with tox -e mutmut (which calls tests/mutmut_workflow.py, which class tests/generate_small_sample.py).
The samples are placed in tests/small_samples

Every sample is the result of a code mutation detected by mutmut which was not covered by any other test (excluding the long running tests).

generate_small_sample.py tries to find a source file with some code which fails with the mutated implementation and removes code from this file with pysource_minimize.

I have also found some issues:

The ExceptHandler cleanup is now an KnownIssue. Before it was verified as an valid mapping, which is not ok.
DELETE/STORE_DEREF had some issues.

I had to reimplement the Deadcode analysis. The problem was that I always tried to reimplement the deadcode analysis of the python interpreter, which did not scale very well. The new implementation replaces the node with an sentinel compiles the source and checks if the sentinel is in the bytecode. The downside is that I had to remove the inverse deadcode check, because the new implementation is a bit more complex ( compiles the whole file for every node to test).

Let me know what you think.

closes #55

15r10nk commented 1 year ago

@alexmojaki it seems that the build is broken because some python versions are missing in the build image.

alexmojaki commented 1 year ago

Change ubuntu-latest to ubuntu-20.04. I've been hitting the same problem in other repos.

15r10nk commented 1 year ago

finds a mutation that causes the normal tests to fail.

no, mutmut finds a mutation that causes the normal tests NOT to fail. generate_small_sample.py searches then for one source file with the specific syntax which let the tests fail (this is in a normal mutmut workflow done by the developer). This condenses a set of thousands of source files down to only a few which are important.

The small_samples make it harder for future contributors to break the code, because a lot of possible scenarios are already covered.

15r10nk commented 1 year ago

summary:

generate_small_sample got much better console output.
generate_small_sample is now also very useful without mutmut.
documented some things in development.md.

I forgot to mention earlier that there are some small_samples which don't work for python<3.11 (see test in test_main.py). I don't know how big of a problem this is.

I hope that there are not many issues left ... it is getting late and I need some sleep now.

Thank you again for your reviews :-)

15r10nk commented 1 year ago

the reason for the failing tests for 3.11 is this https://github.com/alexmojaki/executing/issues/47#issuecomment-1360247694

alexmojaki commented 1 year ago

I'd be fine with the tests having a special case to account for this Python bug.

alexmojaki commented 1 year ago

Does pypy also get stuck locally for you? If not, can you try making pytest verbose?

15r10nk commented 1 year ago

I skipped the test which also needs the increased recursion limit.

I don't know why this is such a big problem. But I think this is not critical, or what do you think?

alexmojaki commented 1 year ago

I don't know why that happens, that's weird. It's not important.

Great work, thanks! Shall I make a PyPI release?

15r10nk commented 1 year ago

I think you don't have to. There should be almost no changes in the behavior.

alexmojaki / executing

Small samples #62