Tox skip missing interpreters

[dHannasch]

This makes Travis pass.

I think in general we probably do want to count it as a pass even if a particular CI host doesn't have e.g. Python 3.8. It's a tricky tradeoff, because it means reporting a pass when we haven't actually tested every env listed in tox.ini. But...

• We certainly do want to test Python 3.8 if possible. So we want it in the list.
• We probably don't want to make a big deal if Python 3.8 isn't available.

Ideally we might specify some Python versions as optional and some as mandatory, but I don't think tox supports that.

[ionelmc]

This would allow misconfigured CI (no python available) to pass wouldn't it?

[dHannasch]

This would allow misconfigured CI (no python available) to pass wouldn't it?

I just tried to run CI on a Docker container with no Python available, and it failed. How exactly could it pass? Is there a way for the CI to have tox without Python?

[ionelmc]

Suppose you have TOXENV=py37,codecov. It will pass because codecov will use any interpreter but it shouldn't cause no tests have run.

[dHannasch]

Suppose you have TOXENV=py37,codecov. It will pass because codecov will use any interpreter but it shouldn't cause no tests have run.

Ah, I see! I didn't know about TOXENV. Hmm.

On the one hand, we want to allow the user to override the list of tox testenvs any way they normally would do that with tox.

On the other hand, they might specify a list that doesn't include any Python versions that the system actually has. The system definitely has some Python --- otherwise tox couldn't run in the first place --- but they might specify an incompatible list of testenvs.

More subtly, they might specify a list of testenvs such that the only ones that can run are the "fake" convenience testenvs that only exist to hook some utilities onto tox and don't actually run the tests.

https://github.com/ionelmc/cookiecutter-pylibrary/blob/master/%7B%7Bcookiecutter.repo_name%7D%7D/tox.ini#L48

We could make a special command that fails the build if the list of tox testenvs doesn't include any envs that actually run the tests (for some whitelist of envs that actually run the tests). But under some circumstances, they might want to run only non-testing envs, and we'd like to make that reasonably easy to do, too.

So really the situation to worry about is if they specify some list of testenvs, less than the full list, including at least one actually-testing testenv, but not any testing testenv for which we have an interpreter available. In that case, that is probably an error, and we should probably fail the build to alert them.

Of course we could trivially have a testenv that will run the tests and will itself use any Python available. The tricky part is that we don't want such a thing to run unless none of the more-specific testing testenvs will run. (Because it will usually be doing redundant work, running the tests on the same Python version that one of the more-specific testing testenvs is running.)

We could have a special testenv that uses any available Python (like codecov), with an extra command in commands_pre that aborts if any of the standard testing testenvs {py35, py36, py37, py38} are in envlist/tox --listenvs (I'd have to check what we should do to actually retrieve the actual runtime list of envs including changes due to TOXENV, but I'm sure there's a way) and are capable of actually running.

This depends on the idea that we can know in advance what the "real" testenvs are. It certainly seems like the names py36 and so forth have special status to tox.

This way, the user could also bypass it if they really truly did intend to not have any actually-testing testenvs run.

I'm not immediately sure how to check whether a testenv is going to run (try to invoke the relavant Python interpreter?) but we could always use depends to run the special env after them --- don't check whether they're going to run, check whether they did in fact run.

Actually, if depends allowed us to specify a dependency on "any" of a list of testenvs rather than on "all", then we'd be immediately done, wouldn't we? We could just specify testenv:canary to depend on any of {py35, py36, py37, py38...} and then the canary will fail unless at least one of its dependencies runs successfully. (The current procedure for changing the list of testenvs already requires the user to re-run bootstrap.py to change .travis.yml, that could also change what the canary requires.) Hmm, patching tox to make skip_missing_interpreters more sophisticated sounded hard since as it stands skip_missing_interpreters does not "cross" testenv boundaries, but patching tox to make depends more sophisticated might not be so hard...only question is how to specify the syntax for "depends on any of these"...

While trying an image that had only Python 3.8, I did notice that report will fail if all of the tests are skipped for lack of Python versions:

 SKIPPED: InterpreterNotFound: python2.7
 SKIPPED: InterpreterNotFound: python3.4
 SKIPPED: InterpreterNotFound: python3.5
 SKIPPED: InterpreterNotFound: python3.6
 SKIPPED: InterpreterNotFound: python3.7
 SKIPPED: InterpreterNotFound: pypy
 SKIPPED: InterpreterNotFound: pypy3
 report create: .tox/report
 report installdeps: coverage
 report installed: coverage==5.0.4
 report run-test: commands[0] | coverage report
 No data to report.
 ERROR: InvocationError for command .tox/report/bin/coverage report (exited with code 1)
 ___________________________________ summary ____________________________________
   clean: commands succeeded
   check: commands succeeded
   docs: commands succeeded
 SKIPPED:  py27: InterpreterNotFound: python2.7
 SKIPPED:  py34: InterpreterNotFound: python3.4
 SKIPPED:  py35: InterpreterNotFound: python3.5
 SKIPPED:  py36: InterpreterNotFound: python3.6
 SKIPPED:  py37: InterpreterNotFound: python3.7
 SKIPPED:  pypy: InterpreterNotFound: pypy
 SKIPPED:  pypy3: InterpreterNotFound: pypy3
 ERROR:   report: commands failed

[ionelmc]

Well since travis/appveyor support has been removed, and in general I want failures to bubble up I'll close this.