[bug] Testing rerun with failure no longer works with pytest 6.1.0

[yuanming-hu]

Describe the bug ti test -Cvr2 -t2 in the CI scripts do longer works with pytest 6.1.0.

To Reproduce

Run ti test -Cvr2 -t2 with pytest 6.1.0

Log/Screenshots

>>> ti test -Cvr2 -t2 && cd python && $PYTHON build.py try_upload && bash <(curl -s https://codecov.io/bash)
*******************************************
**      Taichi Programming Language      **
*******************************************

Docs:   https://taichi.rtfd.io/en/stable
GitHub: https://github.com/taichi-dev/taichi
Forum:  https://forum.taichi.graphics

Running Python tests...

Starting 2 testing thread(s)...
Traceback (most recent call last):
  File "/Users/runner/work/taichi/taichi/bin/ti", line 4, in <module>
    exit(taichi.main())
  File "/Users/runner/work/taichi/taichi/python/taichi/main.py", line 1084, in main
    return cli()
  File "/Users/runner/work/taichi/taichi/python/taichi/main.py", line 27, in wrapper
    result = func(*args, **kwargs)
  File "/Users/runner/work/taichi/taichi/python/taichi/main.py", line 88, in __call__
    return getattr(self, args.command)(sys.argv[2:])
  File "/Users/runner/work/taichi/taichi/python/taichi/main.py", line 929, in test
    ret = TaichiMain._test_python(args)
  File "/Users/runner/work/taichi/taichi/python/taichi/main.py", line 745, in _test_python
    return int(pytest.main(pytest_args))
  File "/Users/runner/.local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "/Users/runner/.local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 319, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/Users/runner/.local/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/runner/.local/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/runner/.local/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/runner/.local/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/Users/runner/.local/lib/python3.7/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse
    config = outcome.get_result()  # type: Config
  File "/Users/runner/.local/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/runner/.local/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/runner/.local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "/Users/runner/.local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 1280, in parse
    self._preparse(args, addopts=addopts)
  File "/Users/runner/.local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 1172, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/Users/runner/.local/lib/python3.7/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/Users/runner/.local/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 105, in load
    module = import_module(match.group('module'))
  File "/Users/runner/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/Users/runner/.local/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 171, in exec_module
    exec(co, module.__dict__)
  File "/Users/runner/.local/lib/python3.7/site-packages/pytest_rerunfailures.py", line 7, in <module>
    from _pytest.resultlog import ResultLog
ModuleNotFoundError: No module named '_pytest.resultlog'
Error: Process completed with exit code 1.
The command "ti test -Cvr2 -t2 && cd python && $PYTHON build.py try_upload && bash <(curl -s https://codecov.io/bash)" exited with 1.

Comments

It works fine with the previous version, i.e., pytest==6.0.2. Therefore I enforced build bots to use this version (https://github.com/taichi-dev/taichi/pull/1900/files) instead for now so that the new version can be released. It's worth figuring out what exactly went wrong.

[Hanke98]

Hi, Yuanming, I reproduced this problem and had a look at the ChangeLog of pytest 6.1.0. In this version, they removed the class ResultLog which is still used by the pytest plugin: pytest-rerunfailures. We use this plugin to re-run some cases, that is to say, if we use the arg of '-r' or '--rerun' in our test command, this problem will occur.

Currently, the developers of pytest-rerunfailures are working with this problem, see this issue and this pr.

Before they fix this problem, I think it is a good idea to just keep the pytest==6.0.2 version !

[yuanming-hu]

Awesome! Thanks for the detailed analysis, @Hanke98! It seems that we can simply wait.

[hugovk]

The pytest-rerunfailures plugin version 9.1.1 has just been released which now supports newest pytest 6.1:

• https://github.com/pytest-dev/pytest-rerunfailures/blob/master/CHANGES.rst#911-2020-09-29

[frostming]

This issue seems no longer valid, the CI is running against -r2(with pytest-rerunfailures enabled) and all work fine.

Please double check and close this, thanks.