4.4: pytest is failing

[kloczek]

I'm trying to package your module as an rpm package. So I'm using the typical build, install and test cycle used on building packages from non-root account.

• "setup.py build"
• "setup.py install --root </install/prefix>"
• "pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

May I ask for help because few units are failing:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-zope-exceptions-4.4-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-zope-exceptions-4.4-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra --import-mode=importlib
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Using --randomly-seed=3257663694
rootdir: /home/tkloczko/rpmbuild/BUILD/zope.exceptions-4.4
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, aspectlib-1.5.2, toolbox-0.5, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, flaky-3.7.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, datadir-1.3.1, regressions-2.2.0, cases-3.6.3, xprocess-0.18.1, black-0.3.12, anyio-3.3.0, asyncio-0.15.1, subtests-0.5.0, isort-2.0.0, hypothesis-6.14.6, mock-3.6.1, profiling-1.7.0, randomly-3.8.0, Faker-8.12.1, nose2pytest-1.0.8, pyfakefs-4.5.1, tornado-0.8.1, twisted-1.13.3, aiohttp-0.3.0
collected 80 items

src/zope/exceptions/tests/test_exceptionformatter.py ...F........................F...................F.............................                                  [ 97%]
src/zope/exceptions/tests/test_log.py ..                                                                                                                             [100%]

================================================================================= FAILURES =================================================================================
__________________________________________________________ TextExceptionFormatterTests.test_extractStack_wo_frame __________________________________________________________

self = <test_exceptionformatter.TextExceptionFormatterTests testMethod=test_extractStack_wo_frame>

    def test_extractStack_wo_frame(self):
        fmt = self._makeOne()
        f = sys._getframe()
        lineno = f.f_lineno
        lines = fmt.extractStack()
        # rather don't assert this here
        # self.assertEqual(len(lines), 10)
>       self.assertEqual(lines[-1], '  Module '
                         'zope.exceptions.tests.test_exceptionformatter, '
                         'line %d, in test_extractStack_wo_frame\n'
                         '    lines = fmt.extractStack()\n' % (lineno + 1))
E       AssertionError: '  Module test_exceptionformatter, line 364, in tes[53 chars]()\n' != '  Module zope.exceptions.tests.test_exceptionforma[75 chars]()\n'
E       -   Module test_exceptionformatter, line 364, in test_extractStack_wo_frame
E       +   Module zope.exceptions.tests.test_exceptionformatter, line 364, in test_extractStack_wo_frame
E       ?          ++++++++++++++++++++++
E             lines = fmt.extractStack()

src/zope/exceptions/tests/test_exceptionformatter.py:367: AssertionError
______________________________________________________ TextExceptionFormatterTests.test_extractStack_wo_frame_w_limit ______________________________________________________

self = <test_exceptionformatter.TextExceptionFormatterTests testMethod=test_extractStack_wo_frame_w_limit>

    def test_extractStack_wo_frame_w_limit(self):
        fmt = self._makeOne(limit=2)
        f = sys._getframe()
        lineno = f.f_lineno
        lines = fmt.extractStack()
        self.assertEqual(len(lines), 3)
>       self.assertEqual(lines[-1], '  Module '
                         'zope.exceptions.tests.test_exceptionformatter, '
                         'line %d, in test_extractStack_wo_frame_w_limit\n'
                         '    lines = fmt.extractStack()\n' % (lineno + 1))
E       AssertionError: '  Module test_exceptionformatter, line 376, in tes[61 chars]()\n' != '  Module zope.exceptions.tests.test_exceptionforma[83 chars]()\n'
E       -   Module test_exceptionformatter, line 376, in test_extractStack_wo_frame_w_limit
E       +   Module zope.exceptions.tests.test_exceptionformatter, line 376, in test_extractStack_wo_frame_w_limit
E       ?          ++++++++++++++++++++++
E             lines = fmt.extractStack()

src/zope/exceptions/tests/test_exceptionformatter.py:378: AssertionError
___________________________________________________________ Test_format_exception.test_format_exception_as_html ____________________________________________________________

self = <test_exceptionformatter.Test_format_exception testMethod=test_format_exception_as_html>

    def test_format_exception_as_html(self):
        # Test for format_exception (as_html=True)
        from zope.exceptions.exceptionformatter import format_exception
        from textwrap import dedent
        import re
        try:
            exec('import')
        except SyntaxError:
            result = ''.join(format_exception(*sys.exc_info(), as_html=True))
        expected = dedent("""\
            <p>Traceback (most recent call last):</p>
            <ul>
            <li>  Module {module}, line ABC, in {fn}<br />
                exec('import')</li>
            </ul><p>  File "&lt;string&gt;", line 1<br />
                import<br />
                     ^<br />
            SyntaxError: invalid syntax<br />
            </p>""").format(
                module='zope.exceptions.tests.test_exceptionformatter',
                fn='test_format_exception_as_html',
        )
        if IS_PY39_OR_GREATER:  # pragma: no cover
            # Python 3.9 moves the pointer after the statement instead to the
            # last character of it:
            expected = expected.replace('^<br />', ' ^<br />')
        # HTML formatter uses Windows line endings for some reason.
        result = result.replace('\r\n', '\n')
        result = re.sub(r'line \d\d\d,', 'line ABC,', result)
        self.maxDiff = None
>       self.assertEqual(expected, result)
E       AssertionError: '<p>T[53 chars]dule zope.exceptions.tests.test_exceptionforma[201 chars]</p>' != '<p>T[53 chars]dule test_exceptionformatter, line ABC, in tes[179 chars]</p>'
E         <p>Traceback (most recent call last):</p>
E         <ul>
E       - <li>  Module zope.exceptions.tests.test_exceptionformatter, line ABC, in test_format_exception_as_html<br />
E       ?              ----------------------
E       + <li>  Module test_exceptionformatter, line ABC, in test_format_exception_as_html<br />
E             exec('import')</li>
E         </ul><p>  File "&lt;string&gt;", line 1<br />
E             import<br />
E                  ^<br />
E         SyntaxError: invalid syntax<br />
E         </p>

src/zope/exceptions/tests/test_exceptionformatter.py:740: AssertionError
============================================================================= warnings summary =============================================================================
.:0
  :0: PytestCollectionWarning: cannot collect test class 'TestingTracebackSupplement' because it has a __init__ constructor (from: src/zope/exceptions/tests/test_exceptionformatter.py)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================= short test summary info ==========================================================================
FAILED src/zope/exceptions/tests/test_exceptionformatter.py::TextExceptionFormatterTests::test_extractStack_wo_frame - AssertionError: '  Module test_exceptionformatter,...
FAILED src/zope/exceptions/tests/test_exceptionformatter.py::TextExceptionFormatterTests::test_extractStack_wo_frame_w_limit - AssertionError: '  Module test_exceptionfo...
FAILED src/zope/exceptions/tests/test_exceptionformatter.py::Test_format_exception::test_format_exception_as_html - AssertionError: '<p>T[53 chars]dule zope.exceptions.t...
================================================================= 3 failed, 77 passed, 1 warning in 0.69s ==================================================================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.

[d-maurer]

Tomasz Kłoczko wrote at 2021-9-7 20:27 -0700:

I'm trying to package your module as an rpm package. So I'm using the typical build, install and test cycle used on building packages from non-root account.

• "setup.py build"
• "setup.py install --root </install/prefix>"
• "pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Most zopefoundation packages have tests designed to be run by zope.testrunner (not pytest).

...

  self.assertEqual(lines[-1], '  Module '

'zope.exceptions.tests.test_exceptionformatter, ' 'line %d, in test_extractStack_wo_frame\n' ' lines = fmt.extractStack()\n' % (lineno + 1)) E AssertionError: ' Module test_exceptionformatter, line 364, in tes[53 chars]()\n' != ' Module zope.exceptions.tests.test_exceptionforma[75 chars]()\n' E - Module test_exceptionformatter, line 364, in test_extractStack_wo_frame

This looks as if pytest starts tests in a different package environment: with pytest, the test module seems to be isolated, with zope.testrunner embedded in its package hierarchy.

Install zope.testrunner and try to run the tests with zope-testrunner --test-path src instead of pytest.

[kloczek]

Most zopefoundation packages have tests designed to be run by zope.testrunner (not pytest). ... > self.assertEqual(lines[-1], ' Module ' 'zope.exceptions.tests.test_exceptionformatter, ' 'line %d, in test_extractStack_wo_frame\n' ' lines = fmt.extractStack()\n' % (lineno + 1)) E AssertionError: ' Module test_exceptionformatter, line 364, in tes[53 chars]()\n' != ' Module zope.exceptions.tests.test_exceptionforma[75 chars]()\n' E - Module test_exceptionformatter, line 364, in test_extractStack_wo_frame This looks as if pytest starts tests in a different package environment: with pytest, the test module seems to be isolated, with zope.testrunner embedded in its package hierarchy. Install zope.testrunner and try to run the tests with zope-testrunner --test-path src instead of pytest.

Thx will try that bnevertheless it would be way better if pytest would be fully supported because it has a lot of plugins which can extend testing procedure without cahgeing anything in test suite or tested code.

[d-maurer]

Tomasz Kłoczko wrote at 2021-9-8 03:41 -0700:

... Thx will try that bnevertheless it would be way better if pytest would be fully supported because it has a lot of plugins which can extend testing procedure without cahgeing anything in test suite or tested code.

zope.testrunner has many built in features - which you select via parameters.

-- Dieter

[kloczek]

zope.testrunner has many built in features - which you select via parameters. … -- Dieter

Yes, however pytest is waaay more advanced and list of plugins is veeery long :) https://docs.pytest.org/en/latest/reference/plugin_list.html

[icemac]

@kloczek The zopefoundation projects have settled on zope.testrunner because it was there before pytest. Projects requiring to be tested using zope.testrunner cannot be tested using pytest because pytest lacks support for at least the feature of layers. It makes no sense to port the tests to pytest, please accept this.

Closing this issue as "won't fix".

[kloczek]

Thing are changeing and now pytestst is available and is far more advanced than what has been developed long time ago.

[icemac]

@kloczek I know pytest and use it in many projects, especially new ones. There is no need to sell pytest to me. Additionally I personally did the conversion from zope.testrunner to pytest for a personal project, it took way more time than expected. So I know both test runners well enough to make an informed decision. And – I have to repeat – the decision is to keep zope.testrunner.

Believe me, you won't be the volunteer to do such a migration, so please do not nag the zopefoundation maintainers: There are more important things to do than changing the test runner.

[icemac]

@kloczek As there is no obvious way to send you a personal message I have to write it here in public: I personally think that most of the issues you created in zopefoundation repositories are not helpful for the projects. It seems that you do want to respect the decisions of the maintainers. Please find a more constructive way to support the zopefoundation projects.

[kloczek]

I think that have possibility to use pytest extensions is really constructive.

[kloczek]

If it is not clear I'm not asking to drop zope.testrunner. Maybe I'm wrong but what I saw I think that it should be possible to have possibility to be able use zope.testrunner and pytest.

[d-maurer]

Tomasz Kłoczko wrote at 2021-9-14 23:53 -0700:

If it is not clear I'm not asking to drop zope.testrunner. Maybe I'm wrong but what I saw I think that it should be possible to have possibility to be able use zope.testrunner and pytest.

Does pytest support test layers? A test layer sets up (and tears down) test infrastructure shared by a set of tests. This significantly reduces test time and is important for Zope related tests.

zope.testrunner supports test layer hierarchies based on Python's class hiararchy. Should pytest does not do so as well, it is in general not adequat for Zope related tests.

[icemac]

No, pytest does not support layers, as I already wrote in https://github.com/zopefoundation/zope.exceptions/issues/21#issuecomment-915844937. There is https://pypi.org/project/gocept.pytestlayer/ to bring layers into pytest but it has its own problems because layers are a concept which is quite different from pytest's fixtures.

Additionally I see no gain in trying to support two test runners. I see only an increased maintenance burden. pytest does not solve any problems in zopefoundation repositories which zope.testrunner didn't already solve.