An easier way to run under multiple Pythons?

[nedbat]

I had a mutant that survived on Python 3, but would have died under Python 2. So I wanted to have mutmut run the tests under both versions.

First I tried:

runner = tox -e py27,py36

but it was much too slow

Next I tried:

runner = .tox/py27/bin/pytest -x && .tox/py36/bin/pytest -x

but it failed (because of no shell I guess?):

  File "/usr/local/virtualenvs/cog/lib/python3.7/site-packages/mutmut/__main__.py", line 579, in time_test_suite
    raise RuntimeError("Tests don't run cleanly without mutations. Test command was: %s\n\nOutput:\n\n%s" % (test_command, '\n'.join(output)))
RuntimeError: Tests don't run cleanly without mutations. Test command was: .tox/py27/bin/pytest -x && .tox/py36/bin/pytest -x

This worked, but is arcane:

runner = sh -c '.tox/py27/bin/pytest -x && .tox/py36/bin/pytest -x'

As an idea, could mutmut support multiple runners, and a mutant survives only if all the runners succeed?

runners = \
    .tox/py27/bin/pytest -x
    .tox/py36/bin/pytest -x

[boxed]

Yea, tox/pip is extremely slow on startup. I've suffered in other projects because of this :(

I like the idea of multiple runners! Normally I just mark python 2 as no mutate. It's dead soon enough anyway.

[nklapste]

@boxed We could add an option (maybe --shell-runner) to invoke the subprocess.Popen() call within popen_streaming_output with the shell=True kwarg. This should allow the .tox/py27/bin/pytest -x && .tox/py36/bin/pytest -x to execute without failure as it executes the command through the systems shell.

The primary security issues of running subprocess.Popen() with the shell=True kwarg (i.e. shell injection) are not present in our use case as the person running mutmut should trust their own input.

This option could also have use in cases where a runner needs the system shell environment (e.g. working with environment variables, silencing a return code, playing with pipes, ect...) to effectively execute for a mutmut test run.

However, there is still benefit in invoking subprocess.Popen() with the shell=False kwarg as it limits the overhead of spawning a subprocess.

[boxed]

Is that overhead really significant for our use case though?

[nklapste]

I would assume the overhead of starting up a subshell would be insignificant for most cases (maybe just set shell=True and nix adding a --shell-runner arg). Python itself already has a slow startup (relatively), and starting up another instance of sh isn't going hurt.

[boxed]

Tried this in ipython:

In [3]: %timeit check_output(['ls'])                                                                                                       
5.58 ms ± 55.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [4]: %timeit check_output('ls', shell=True)                                                                                             
8.24 ms ± 191 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Seems pretty insignificant, so yea, using shell=True seems like a reasonable thing to do.