option to fail fast

[jtiai]

If having configure multiple environments and running them with plain tox-command if environment in the middle fails, rest of the environments are run.

Expected that if one environment fails, rest aren't executed.

Sample tox.ini:

[tox]
envlist = lint,test,build
skipsdist = True

[testenv]
whitelist_externals = sh

[testenv:lint]
commands =
    sh -c "exit 0"

[testenv:test]
commands =
    sh -c "exit 1"

[testenv:build]
commands =
    sh -c "exit 0"

pip list:

pip (9.0.1)
pkg-resources (0.0.0)
pluggy (0.4.0)
py (1.4.34)
setuptools (36.2.7)
tox (2.7.0)
virtualenv (15.1.0)
wheel (0.29.0)

[RonnyPfannschmidt]

for tox running everything is expectedbehaviour

[The-Compiler]

That seems like expected behaviour to me. If e.g. tests with python 2 fail, I'd still like to run tests with python 3.

[obestwalter]

Hi @jtiai as already said: this is how it's supposed to be :)

You could get to the wanted behaviour by doing something like this on the command line (same semantics on POSIX and Windows):

tox -e env1 && tox -e env2 && tox -e env3

If any of those runs fail the rest of the chain will not be executed anymore.

On CI systems you usually have the concept of stages that are executed serially and if one stage fails the whole plan fails and nothing else is executed. So there you would create a stage for each of the tox environments that depend on each other.

[jtiai]

I got the impression that tox stops the execution from the docs: http://tox.readthedocs.io/en/latest/config.html#confval-ignore_errors=True|False(default)

So maybe it's just matter of clarify in the documentation that only current environment execution is aborted, but rest of the environments defined will be executed regardless state of the others.

[The-Compiler]

@jtiai Do you have any suggestion on how this could be clarified? It already says "will abort execution of commands for that environment.".

[jtiai]

Last paragraph states:

Tox >= 2.0 will abort on an error by default, which is safer and more typical of CI and command execution tools, as it doesn’t make sense to run tests if installing some prerequisite failed and it doesn’t >make sense to try to deploy if tests failed.

So that's where I got impression that tox stops on error.

Not sure about wording but that part should somehow emphasize the fact that only current environment execution is stopped and if there are more defined, either in envlist configuration or via -e switch from the cli rest will be executed.

[gaborbernat]

I think we only need to point out that that abort is active inside the commands :+1: not for the environments :+1:

[apomelov]

Take a look at ignore_outcome's doc as well. Now it says:

If set to True a failing result of this testenv will not make tox fail, only a warning will be produced. default: False

Really expecting tox to fail by default after some testenv fails.

[obestwalter]

o.k. that surprises me, but I get now where you are coming from, so we definitely need to make the docs clearer in those areas. Thanks!

[obestwalter]

Hi @apomelov, this also contains the term "testenv" which means that it only means that the testenv won't fail, but I get it. I think what is missing is a bit of a high level overview that is describing a tox run and everything that is involved and properly introduces the terminology.

[obestwalter]

If set to True a failing result of this testenv will not make tox fail, only a warning will be produced. default: False

And reading it again I realize the misunderstanding is a different one here ... sorry @apomelov.

What this means is that the whole tox run will not be regarded as a failure, if the testenv fails if that flag is set. It will still run all the environments. So this should be explicitly added there I guess.

[obestwalter]

Next thought is obviously to introduce a flag that tox should stop running if one testenv fails, but I see trouble that way (e.g. what is detox supposed to do, which will end up in core at some point).

[RonnyPfannschmidt]

well, quite simple, once one fails, don't schedule new ones to get started

[obestwalter]

well, quite simple, once one fails, don't schedule new ones to get started

This falls down with detox though. I still think that tox should just run all the envs you tell it to run. If you want to run them staged then address them like I mentioned above.

[RonnyPfannschmidt]

thats precisely why i used the word schedule instead of run, i didn't want to suggest to cancel already running ones, but i did want to suggest to simply stop scheduling additional executions after the first failure

[jan-matejka]

This would be really useful for running tox within git-bisect

[KrashLeviathan]

I would find this option useful as well. I'd like my linter, unit tests, and end-to-end tests in different environments, and there should be an option to stop early if the linter or unit tests fail to avoid running the long e2e tests. E.g. something like this: tox -e lint,py36,py36-e2e --stop-on-fail

[ssbarnea]

For development perspective I would find this very useful because I don't want to scroll lots of screens in order to see what failed. I just want to get to the failure as quick as possible, fix it and run again.

I can understand that for CI the desire may be different but if this feature would be controlable using both tox.ini entry and a TOX_ variable it would be ideal, as I could just define the variable for my development machine and not have to touch the codebase.

[igalic]

right now i'm dong a big refactor on a big python project i move one function into place, run tox, wait 3 minutes, and then scroll about 18000 lines to see what next to do…

on the one hand, this leaves a lot of time for contemplation: why am i doing this? why am i not doing something i get paid for? why didn't i listen to my parents and learn something sensible like goat herding? on the other hand it would be quite useful to get this done quickly and have less existential angst.

[gaborbernat]

@igalic if you're refactoring you don't need to run the entire test suite all the time. You can use the -e option to select just one environment (e.g. one interpreter one test suite only, ignore linter, etc). You can also use the {posargs}} to forward arguments to your test suite runner to further decrease the number of tests to run needlessly.

[obestwalter]

Hi @igalic, does that mean that in order to find out what is going wrong, you always have to run all toxenvs? Otherwise I would also go with the suggestion from Bernát or even activate the toxenv and run specific tests there until you are happy and only then run the whole suite on all envs to see what broke (Ideally in parallel on your CI or with detox locally to speed things up).

As issue is now a 3.6 milestone this might be added as an advanced feature after all, but from my perspective this is really not necessary.

I think that CI should be responsible for this kind of thing (e.g. in the case of Travis via build stages, which call the toxenvs in the order they are needed, failing whenever a stage fails). As I didn't actively follow the development over the last few months I might have missed a few decisions and maybe we start to turn tox into a complete CI system, but until then I don't think this is not tox' job. I think I better go read up what else I missed now :)

[obestwalter]

Sorry my bad. the milestone is about the documentation part of this sigh of relief.

[igalic]

indeed, running only one of the envs greatly cut down on waiting and scrolling time, and i was able to make some actual progress!

thank you all for the advice!

[jdahlin]

I would love to see this feature as well, we have thousands of tests and it's quite hard to scroll up in the list to find the previous failures.

[gaborbernat]

@jdahlin those thousand test running are handled by your test runner, not by tox; so the test runner provides the availability to stop, not tox. All we can do is that if py37 failed we don't start py36 too... but that's tox environment count, not test count.

[ssbarnea]

Any chance of making this happen? I got others complaining about the same issues. With Travis resource limitations we do often want to group multiple environments inside a single job. Using the workaround of calling tox multiple times for each environment works would required an almost complete rewrite of some complex travis.yaml files.

See https://github.com/ansible/molecule/pull/2257#pullrequestreview-281431764 where one colleague opposed consolidation for the same reasons.

Maybe this does not appear as a big issue for small projects, but if you end-up having 27 environments like we have on molecule, you realize that this bug becomes a real problem.

[gaborbernat]

Chances are that yes, within tox 4 I would say, but not likely to happen before end of year.

[paunovic]

+1 for this feature. We utilize Tox in our CI/CD pipeline and it would greatly shorten testing time if there was an option for Tox to fail fast on error.

[boxed]

I would also like to pile on to this. I was quite surprised that this wasn't an available option.

I agree that it shouldn't be the default, just as it's not the default for pytest, but it's a useful option.

[paunovic]

Hi, any ETA on this?

[gaborbernat]

Still caught up in trying to stabilise virtualenv, I'd say no chance before end of year 👍for this to happen with tox 4.

[saroad2]

Do you need any help on implementing this? I really need this ability for my project and would love to give a hand on it :)

[gaborbernat]

If someone wants to implement it against current master I can help review it and land it 🤷‍♂️

[nkpro2000sr]

If it needs to only stop some envs if one envs fails
and not to stop other envs

Sample Situation :

If we need to only stop all other PyPys if any one of PyPy fails, still continue other envs and only stop other CPythons if any one of CPython fails.

Maybe this approach helps :smiley:

[tox]
envlist =  {py27,py35}-batch1,{pypy2,pypy3}-batch2

[testenv:batch1]
stoponfail = True

[testenv:batch2]
stoponfail = True

[frenzymadness]

Until this will be implemented, I'm using for loop for it like:

for env in `tox -l`; do echo $env; tox -e $env || break; done

[dqalombardi]

Any word on this issue ?

For my use-case, there are just some testenv's whose success is naturally prerequisite to the execution of any others. For instance, if black fails because there is a syntax error in a source file, I cannot imagine a use-case in which the user would want to continue on to run linters, type-checkers, unit tests, etc. on broken source code. (I would actually say that formatters, linters, and static type-checkers are all naturally prerequisite to the execution of unit tests.)

Just to add my two cents, I think the following implementation would be great (where stop_on_fail indicates that if the testenv fails, then tox should abort).

[tox]
requires =
    tox>=4
env_list =
    format
    py312

[testenv:format]
description = run formatters on src code
stop_on_fail = true 
deps =
    black
commands =
    black src

[testenv]
description = run test code
deps =
    pytest
commands =
    pytest test

Are there any objections or known technical pain-points for such an implementation ? Or is it more a matter of bandwidth at the moment ?

[gaborbernat]

Any word on this issue ?

Yes, PR is welcome. We do not plan to do this ourselves.