Improve the output when a tox env failed

[obestwalter]

When a toxrun fails it prints something like:

ERROR: InvocationError for command '/path/to/.tox/pypy/bin/pytest tests' (exited with code 1)

My experience with new users is that they are usually confused about this and think that there might have been a problem with tox itself and they don't really know what to make of this.

I think this could be improved by phrasing this differently and presenting the information in an easier to grasp format. Looking at the acutal information we have:

ERROR                      # o.k.
InvocationError            # why show an internal exception name here?
command                    # o.k.
exit code (new since 3.0)  # o.k.

How about something like:

Error in <envname> <stage name>. 
    <command> returned exit code <code>

e.g.

Error in py27-tests commands. 
    '/path/to/.tox/pypy/bin/pytest tests' failed with exit code 1

[aspiers]

Furthermore, tox hides what would have been STDERR in at least one case, which requires the user to re-run the failing command in the hope that it will reproduce the failure, but this time without hiding STDERR. For example I'm currently running:

$ tox -e py27 -vvv
  removing /home/adam/git/myproject/.tox/log
using tox.ini: /home/adam/git/myproject/tox.ini
using tox-2.9.1 from /usr/lib/python3.4/site-packages/tox/__init__.py
skipping sdist step
py27 start: getenv /home/adam/git/myproject/.tox/py27
determining /home/adam/.tox/distshare/git-deps-*.zip
py27 reusing: /home/adam/git/myproject/.tox/py27
py27 finish: getenv after 0.08 seconds
py27 start: developpkg /home/adam/git/myproject
  /home/adam/git/myproject$ /home/adam/git/myproject/.tox/py27/bin/python /home/adam/git/myproject/setup.py --name
py27 finish: developpkg after 0.20 seconds
py27 start: envreport 
setting PATH=/home/adam/git/myproject/.tox/py27/bin:/home/adam/.rvm/gems/ruby-2.2.1/bin:/home/adam/.rvm/gems/ruby-2.2.1@global/bin:/home/adam/.rvm/rubies/ruby-2.2.1/bin:/home/adam/.pyenv/shims:/home/adam/.pyenv/bin:/home/adam/.local/bin:/home/adam/sbin:/home/adam/bin:/usr/local/bin:/usr/X11R6/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/adam/.rvm/bin
  /home/adam/git/myproject$ /home/adam/git/myproject/.tox/py27/bin/pip freeze >/home/adam/git/myproject/.tox/py27/log/py27-1.log
py27 installed: attrs==18.1.0,cffi==1.11.5,click==6.7,Flask==1.0.2,funcsigs==1.0.2,git-deps==1.0.0,itsdangerous==0.24,Jinja2==2.10,MarkupSafe==1.0,more-itertools==4.1.0,ostruct==2.1.2,pluggy==0.6.0,py==1.5.3,pycparser==2.18,pygit2==0.27.0,pytest==3.5.1,six==1.11.0,Werkzeug==0.14.1
py27 finish: envreport after 0.25 seconds
__________________________________________________________________ summary ___________________________________________________________________
ERROR:   py27: InvocationError: '/home/adam/git/myproject/.tox/py27/bin/python /home/adam/git/myproject/setup.py --name'

If I re-run the failed command manually, I get

Traceback (most recent call last):
  File "/home/adam/git/myproject/setup.py", line 23, in <module>
    setup_package()
  File "/home/adam/git/myproject/setup.py", line 19, in setup_package
    use_pyscaffold=True)
  File "/home/adam/git/myproject/.tox/py27/lib/python2.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib64/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/home/adam/git/myproject/.tox/py27/lib/python2.7/site-packages/setuptools/dist.py", line 363, in __init__
    _Distribution.__init__(self, attrs)
  File "/usr/lib64/python2.7/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/home/adam/git/myproject/.tox/py27/lib/python2.7/site-packages/setuptools/dist.py", line 519, in finalize_options
    ep.load()(self, ep.name, value)
  File "/home/adam/git/myproject/.eggs/PyScaffold-2.5.6-py2.7.egg/pyscaffold/integration.py", line 102, in pyscaffold_keyword
    check_setuptools_version()
  File "/home/adam/git/myproject/.eggs/PyScaffold-2.5.6-py2.7.egg/pyscaffold/utils.py", line 211, in check_setuptools_version
    "Your setuptools version is too old (<12).\n"
RuntimeError: Your setuptools version is too old (<12).
Use `pip install -U setuptools` to upgrade.

and only then does some googling reveal that the true problem is in PyScaffold. It would be a far better user experience if STDERR wasn't hidden, or if it was at least captured to a log file, with STDOUT pointing the user at that file.

[obestwalter]

Hi @aspiers, I agree. I actually suspect this might be a regression. Could you run this with version of tox e.g. <2.6 and see if this is still swallowed? Thank you.