checking sanity of environment missing?

[tomato42]

I've executed cosmic-ray against python-ecdsa (with pytest) and was surprised that it achieved 100% kill rate.

but upon closer inspection, this was caused by the fact that pytest didn't like the setup:

============================= test session starts ==============================
platform linux2 -- Python 2.7.16, pytest-3.6.4, py-1.5.4, pluggy-0.6.0
rootdir: /tmp/tmpi_l_uc76/repo, inifile:
plugins: hypothesis-3.66.11
collected 1820 items / 3 errors

==================================== ERRORS ====================================
___________________ ERROR collecting src/ecdsa/test_ecdsa.py ___________________
import file mismatch:
imported module 'ecdsa.test_ecdsa' has this __file__ attribute:
  /home/hkario/dev/python-ecdsa/src/ecdsa/test_ecdsa.py
which is not the same as the test file we want to collect:
  /tmp/tmpi_l_uc76/repo/src/ecdsa/test_ecdsa.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
_______________ ERROR collecting src/ecdsa/test_ellipticcurve.py _______________
import file mismatch:
imported module 'ecdsa.test_ellipticcurve' has this __file__ attribute:
  /home/hkario/dev/python-ecdsa/src/ecdsa/test_ellipticcurve.py
which is not the same as the test file we want to collect:
  /tmp/tmpi_l_uc76/repo/src/ecdsa/test_ellipticcurve.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
_______________ ERROR collecting src/ecdsa/test_numbertheory.py ________________
import file mismatch:
imported module 'ecdsa.test_numbertheory' has this __file__ attribute:
  /home/hkario/dev/python-ecdsa/src/ecdsa/test_numbertheory.py
which is not the same as the test file we want to collect:
  /tmp/tmpi_l_uc76/repo/src/ecdsa/test_numbertheory.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
!!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 3 error in 1.69 seconds ============================

with the following .toml file:

# config.toml
[cosmic-ray]
module-path = "src/ecdsa/"
python-version = ""
timeout = 30
exclude-modules = ["*/test*", "*/_version.py", "test*.py", "_version.py"]
test-command = "pytest"
execution-engine.name = "local"

[cosmic-ray.cloning]
method = 'copy'
commands = []

maybe the runner should validate the sanity of environment (by executing the test suite from the new location but with no changes) before declaring the mutants killed?

this is with cosmic-ray==5.6.1

[boxed]

Mutmut does this precisely to avoid this. It's a good sanity check!

[abingham]

That's what the recently re-added baseline command is for. It runs your suite over unmutated code; if there are any test failures, it gives a return code of 1, otherwise 0.

Does this give you what you need?

[tomato42]

That's what the recently re-added baseline command is for.

where it's documented?

[abingham]

where it's documented?

Unfortunately it's not currently mentioned in the documentation. It was introduced in a flurry of messy PRs that I barely had time to process (hence some of the recent quality issues!)

You can run cosmic-ray baseline -h for some insight. And of course I'd be delighted if you (or anyone) wants to add a little blurb about baseline to the docs.