search

sixty-north / cosmic-ray

Mutation testing for Python
MIT License
565 stars 57 forks source link

Get CR working on coveragepy #426

Closed abingham closed 3 years ago

abingham commented 5 years ago

In #246 @nedbat reported some issues getting CR to run on coverage, and I wanted to get the discussion into its own thread.

abingham commented 5 years ago

@nedbat Try this TOML config:

[cosmic-ray]
module-path = "coverage"
python-version = ""
timeout = 240
excluded-modules = []
test-command = "{python-executable} -m pytest tests"

[cosmic-ray.execution-engine]
name = "local"

[cosmic-ray.cloning]
method = "copy"
commands = [
    "{python-executable} setup.py install",
    "{python-executable} -m pip install -r requirements/pytest.pip"
]

This seems to be working on my machine, though it's not through the 8000-ish tests yet. To avoid the ImportMismatchError you were seeing, you'll need to delete all of the *.pyc files in the tree before running cosmic-ray exec. Apparently path information in the pycs is upsetting pytest.

If you want to take CR for a spin without running all of the mutations (it looks like it'll take several hours), you can change module-path to just one module, e.g. module-path = coverage/bytecode.py or something. Similarly, you could limit the tests that are run with e.g. test-command = "pytest tests/test_annotate.py".

abingham commented 5 years ago

Running CR on coverage turned up some issues related to cloning the source repo and creating a new virtual env. I think I've addressed these issues on the fix-cloning branch, so you might give that a try.

However, I see failures when I run the tests on the master branch of coveragepy. This means that we can't really apply CR to it. CR uses test failures to signal that a mutant has been "killed", so it requires that the test suite passes on unmutated code. As I mentioned earlier, you can get some benefit by only running a subset of tests that do pass, but right now it seems that CR can't be applied to the entire coverage code base / test suite.