search

zopefoundation / zope.testrunner

This package provides a flexible test runner with layer support.
https://zopetestrunner.readthedocs.io/
Other
2 stars 23 forks source link

5.5: pytest is failing #137

Closed kloczek closed 2 years ago

kloczek commented 2 years ago

I know that you don't like pytest however I see really good value in use pyterstbecause it is way more actively developed and flexible so .. Despite that fact that there asome diffrewnces in our approaches there are few few obviouse issuess which blocks "test as installed" mthodology. Hopefully we will meet somemewhere in the middle this time 😄

================================================================================== ERRORS ================================================================================== ____ ERROR collecting src/zope/testrunner/tests/test_digraph.py ____ ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/test_digraph.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: src/zope/testrunner/tests/test_digraph.py:3: in from testrunner.digraph import DiGraph E ModuleNotFoundError: No module named 'testrunner' ____ ERROR collecting src/zope/testrunner/tests/test_doctest.py ____ ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/test_doctest.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: src/zope/testrunner/tests/test_doctest.py:27: in from testrunner.util import uses_refcounts E ModuleNotFoundError: No module named 'testrunner' __ ERROR collecting src/zope/testrunner/tests/test_logsupport.py ___ ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/test_logsupport.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: src/zope/testrunner/tests/testlogsupport.py:11: in from ..logsupport import Logging E ModuleNotFoundError: No module named 'src.zope.testrunner.logsupport'; 'src.zope.testrunner' is not a package ERROR collecting src/zope/testrunner/tests/testthreadsupport.py ____ ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/test_threadsupport.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: src/zope/testrunner/tests/test_threadsupport.py:18: in from testrunner.threadsupport import current_frames, enumerate E ModuleNotFoundError: No module named 'testrunner' ========================================================================= short test summary info ========================================================================== ERROR src/zope/testrunner/tests/test_digraph.py ERROR src/zope/testrunner/tests/test_doctest.py ERROR src/zope/testrunner/tests/test_logsupport.py ERROR src/zope/testrunner/tests/test_threadsupport.py !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ============================================================================ 4 errors in 0.40s =============================================================================

This can be fixed by patch like below
```patch
--- a/src/zope/testrunner/tests/test_threadsupport.py~  2022-06-24 15:27:21.000000000 +0000
+++ b/src/zope/testrunner/tests/test_threadsupport.py   2022-06-25 22:33:50.906655755 +0000
@@ -15,7 +15,7 @@
 from time import sleep
 from unittest import TestCase, skipUnless

-from ..threadsupport import current_frames, enumerate
+from zope.testrunner.threadsupport import current_frames, enumerate

 class ThreadMixin(object):
--- a/src/zope/testrunner/tests/test_doctest.py~        2022-06-24 15:27:21.000000000 +0000
+++ b/src/zope/testrunner/tests/test_doctest.py 2022-06-25 22:32:52.079645588 +0000
@@ -24,7 +24,7 @@

 from zope.testing import renormalizing

-from ..util import uses_refcounts
+from zope.testrunner.util import uses_refcounts

 # separated checkers for the different platform,
 # because it s...s to maintain just one
--- a/src/zope/testrunner/tests/test_digraph.py~        2022-06-24 15:27:21.000000000 +0000
+++ b/src/zope/testrunner/tests/test_digraph.py 2022-06-25 22:30:52.445624917 +0000
@@ -1,6 +1,6 @@
 from unittest import TestCase

-from ..digraph import DiGraph
+from zope.testrunner.digraph import DiGraph

 class Tests(TestCase):
--- a/src/zope/testrunner/tests/test_logsupport.py~     2022-06-24 15:27:21.000000000 +0000
+++ b/src/zope/testrunner/tests/test_logsupport.py      2022-06-25 22:38:07.316695993 +0000
@@ -8,7 +8,7 @@
 from unittest import TestCase
 from warnings import catch_warnings

-from ..logsupport import Logging
+from zope.testrunner.logsupport import Logging

 setup_logging = Logging.global_setup

src/zope/testrunner/tests/test_digraph.py ...... [ 13%] src/zope/testrunner/tests/test_doctest.py . [ 15%] src/zope/testrunner/tests/test_filter.py ......... [ 35%] src/zope/testrunner/tests/test_find.py FFFF [ 44%] src/zope/testrunner/tests/test_logsupport.py ... [ 51%] src/zope/testrunner/tests/test_runner.py ........F...... [ 84%] src/zope/testrunner/tests/test_subunit.py . [ 86%] src/zope/testrunner/tests/test_threadsupport.py ...... [100%]

================================================================================= FAILURES ================================================================================= __ TestUniqueness.test_DuplicateTestIDError_message_contains_all_test_ids __

self = 

def setUp(self):
    super(TestUniqueness, self).setUp()
    suites = [
      doctest.DocFileSuite('testrunner-ex/sampletests.rst'),

doctest.DocFileSuite('testrunner-ex/sampletests.rst'), doctest.DocFileSuite('testrunner-ex/sampletestsl.rst'), doctest.DocFileSuite('testrunner-ex/sampletestsl.rst'), ]

src/zope/testrunner/tests/test_find.py:39:

/usr/lib64/python3.8/doctest.py:2507: in DocFileSuite suite.addTest(DocFileTest(path, **kw)) /usr/lib64/python3.8/doctest.py:2429: in DocFileTest doc, path = _load_testfile(path, package, module_relative, /usr/lib64/python3.8/doctest.py:227: in _load_testfile file_contents = package.loader.get_data(filename)

self = <_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3bb78d8d90> pathname = '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests.rst'

def get_data(self, pathname: Union[str, bytes]) -> bytes:
    """Optional PEP302 get_data API."""
  with open(pathname, "rb") as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests.rst'

/usr/lib/python3.8/site-packages/pytest/assertion/rewrite.py:273: FileNotFoundError ____ TestUniqueness.test_find_tests_raises_error_if_duplicates_found __

self = 

def setUp(self):
    super(TestUniqueness, self).setUp()
    suites = [
      doctest.DocFileSuite('testrunner-ex/sampletests.rst'),

doctest.DocFileSuite('testrunner-ex/sampletests.rst'), doctest.DocFileSuite('testrunner-ex/sampletestsl.rst'), doctest.DocFileSuite('testrunner-ex/sampletestsl.rst'), ]

src/zope/testrunner/tests/test_find.py:39:

/usr/lib64/python3.8/doctest.py:2507: in DocFileSuite suite.addTest(DocFileTest(path, **kw)) /usr/lib64/python3.8/doctest.py:2429: in DocFileTest doc, path = _load_testfile(path, package, module_relative, /usr/lib64/python3.8/doctest.py:227: in _load_testfile file_contents = package.loader.get_data(filename)

self = <_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3bb78d8d90> pathname = '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests.rst'

def get_data(self, pathname: Union[str, bytes]) -> bytes:
    """Optional PEP302 get_data API."""
  with open(pathname, "rb") as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests.rst'

/usr/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:273: FileNotFoundError _ TestUniqueness.test_tests_from_suite_ignores_duplicate_ids_if_option_notset

self = 

def setUp(self):
    super(TestUniqueness, self).setUp()
    suites = [
      doctest.DocFileSuite('testrunner-ex/sampletests.rst'),

doctest.DocFileSuite('testrunner-ex/sampletests.rst'), doctest.DocFileSuite('testrunner-ex/sampletestsl.rst'), doctest.DocFileSuite('testrunner-ex/sampletestsl.rst'), ]

src/zope/testrunner/tests/test_find.py:39:

/usr/lib64/python3.8/doctest.py:2507: in DocFileSuite suite.addTest(DocFileTest(path, **kw)) /usr/lib64/python3.8/doctest.py:2429: in DocFileTest doc, path = _load_testfile(path, package, module_relative, /usr/lib64/python3.8/doctest.py:227: in _load_testfile file_contents = package.loader.get_data(filename)

self = <_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3bb78d8d90> pathname = '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests.rst'

def get_data(self, pathname: Union[str, bytes]) -> bytes:
    """Optional PEP302 get_data API."""
  with open(pathname, "rb") as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests.rst'

/usr/lib/python3.8/site-packages/pytest/assertion/rewrite.py:273: FileNotFoundError ____ TestUniqueness.test_tests_from_suite_records_duplicate_test_ids __

self = 

def setUp(self):
    super(TestUniqueness, self).setUp()
    suites = [
      doctest.DocFileSuite('testrunner-ex/sampletests.rst'),

doctest.DocFileSuite('testrunner-ex/sampletests.rst'), doctest.DocFileSuite('testrunner-ex/sampletestsl.rst'), doctest.DocFileSuite('testrunner-ex/sampletestsl.rst'), ]

src/zope/testrunner/tests/test_find.py:39:

/usr/lib64/python3.8/doctest.py:2507: in DocFileSuite suite.addTest(DocFileTest(path, **kw)) /usr/lib64/python3.8/doctest.py:2429: in DocFileTest doc, path = _load_testfile(path, package, module_relative, /usr/lib64/python3.8/doctest.py:227: in _load_testfile file_contents = package.loader.get_data(filename)

self = <_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3bb78d8d90> pathname = '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests.rst'

def get_data(self, pathname: Union[str, bytes]) -> bytes:
    """Optional PEP302 get_data API."""
  with open(pathname, "rb") as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests.rst'

/usr/lib/python3.8/site-packages/pytest/assertion/rewrite.py:273: FileNotFoundError ____ TestWarnings.test_warning_filterdefault ____

self = 

def test_warning_filter_default(self):
    # When we run tests, we run them with a 'default' simplefilter.
    # Note that this test will fail if PYTHONWARNINGS is set,
    # or a -W option was given, so we skip it
    import warnings
    # Save the current filters, ignoring the compiled regexes,
    # which can't be compared.
    old_filters = [(f[0], f[2], 4) for f in warnings.filters]
    with warnings.catch_warnings():
        # Set up just like the runner does
        warnings.simplefilter('default')
        warnings.filterwarnings('module',
                                category=DeprecationWarning,
                                message=r'Please use assert\w+ instead.')
        new_filters = [(f[0], f[2], 4) for f in warnings.filters]

    # For some reason, catch_warnings doesn't fully reset things,
    # and we wind up with some duplicate entries in new_filters
  self.assertEqual(set(old_filters), set(new_filters))

E AssertionError: Items in the second set but not the first: E ('module', <class 'DeprecationWarning'>, 4) E ('default', <class 'Warning'>, 4)

src/zope/testrunner/tests/test_runner.py:207: AssertionError ========================================================================= short test summary info ========================================================================== FAILED src/zope/testrunner/tests/test_find.py::TestUniqueness::test_DuplicateTestIDError_message_contains_all_test_ids - FileNotFoundError: [Errno 2] No such file or dir... FAILED src/zope/testrunner/tests/test_find.py::TestUniqueness::test_find_tests_raises_error_if_duplicates_found - FileNotFoundError: [Errno 2] No such file or directory:... FAILED src/zope/testrunner/tests/test_find.py::TestUniqueness::test_tests_from_suite_ignores_duplicate_ids_if_option_not_set - FileNotFoundError: [Errno 2] No such file ... FAILED src/zope/testrunner/tests/test_find.py::TestUniqueness::test_tests_from_suite_records_duplicate_test_ids - FileNotFoundError: [Errno 2] No such file or directory:... FAILED src/zope/testrunner/tests/test_runner.py::TestWarnings::test_warning_filter_default - AssertionError: Items in the second set but not the first: ======================================================================= 5 failed, 40 passed in 1.01s =======================================================================


In this case `src/zope/testrunner/tests/test_find.py` units are failing because there is no src/zope/testrunner/tests/testrunner-ex/ files.
kloczek commented 2 years ago

My small fault annd update .. I just found that when I've imported spec file from fedora in that spec was remove content of src/zope/testrunner/tests/testrunner-ex/ and I just found that content of that directoty still contains python 2.x code. Py2 is alredy +2y EOSed.

kloczek commented 2 years ago

After remove "rm -rf src/zope/testrunner/tests/testrunner-ex*" Final result of the pytest

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-zope-testrunner-5.5-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-zope-testrunner-5.5-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra --import-mode=importlib src/zope/testrunner/tests
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5
collected 85 items

src/zope/testrunner/tests/test_digraph.py ......                                                                                                                     [  7%]
src/zope/testrunner/tests/test_doctest.py .                                                                                                                          [  8%]
src/zope/testrunner/tests/test_filter.py .........                                                                                                                   [ 18%]
src/zope/testrunner/tests/test_find.py ....                                                                                                                          [ 23%]
src/zope/testrunner/tests/test_logsupport.py ...                                                                                                                     [ 27%]
src/zope/testrunner/tests/test_runner.py ........F......                                                                                                             [ 44%]
src/zope/testrunner/tests/test_subunit.py .                                                                                                                          [ 45%]
src/zope/testrunner/tests/test_threadsupport.py ......                                                                                                               [ 52%]
src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_none_test.py F                                                                                           [ 54%]
src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py .........EEE.                                                                                [ 69%]
src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py .........EEE.                                                                                  [ 84%]
src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py .........EEE.                                                                                        [100%]

================================================================================== ERRORS ==================================================================================
________________________________________________________________________ ERROR at setup of test_y0 _________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py, line 76
  def test_y0(self):
E       fixture 'self' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py:76
________________________________________________________________________ ERROR at setup of test_x0 _________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py, line 84
  def test_x0(self):
E       fixture 'self' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py:84
________________________________________________________________________ ERROR at setup of test_z1 _________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py, line 92
  def test_z1(self):
E       fixture 'self' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py:92
________________________________________________________________________ ERROR at setup of test_y0 _________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py, line 76
  def test_y0(self):
E       fixture 'self' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py:76
________________________________________________________________________ ERROR at setup of test_x0 _________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py, line 84
  def test_x0(self):
E       fixture 'self' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py:84
________________________________________________________________________ ERROR at setup of test_z1 _________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py, line 92
  def test_z1(self):
E       fixture 'self' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py:92
________________________________________________________________________ ERROR at setup of test_y0 _________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py, line 76
  def test_y0(self):
E       fixture 'self' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py:76
________________________________________________________________________ ERROR at setup of test_x0 _________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py, line 84
  def test_x0(self):
E       fixture 'self' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py:84
________________________________________________________________________ ERROR at setup of test_z1 _________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py, line 92
  def test_z1(self):
E       fixture 'self' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/tkloczko/rpmbuild/BUILD/zope.testrunner-5.5/src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py:92
================================================================================= FAILURES =================================================================================
_________________________________________________________________ TestWarnings.test_warning_filter_default _________________________________________________________________

self = <src.zope.testrunner.tests.test_runner.TestWarnings testMethod=test_warning_filter_default>

    def test_warning_filter_default(self):
        # When we run tests, we run them with a 'default' simplefilter.
        # Note that this test will fail if PYTHONWARNINGS is set,
        # or a -W option was given, so we skip it
        import warnings
        # Save the current filters, ignoring the compiled regexes,
        # which can't be compared.
        old_filters = [(f[0], f[2], 4) for f in warnings.filters]
        with warnings.catch_warnings():
            # Set up just like the runner does
            warnings.simplefilter('default')
            warnings.filterwarnings('module',
                                    category=DeprecationWarning,
                                    message=r'Please use assert\w+ instead.')
            new_filters = [(f[0], f[2], 4) for f in warnings.filters]

        # For some reason, catch_warnings doesn't fully reset things,
        # and we wind up with some duplicate entries in new_filters
>       self.assertEqual(set(old_filters), set(new_filters))
E       AssertionError: Items in the second set but not the first:
E       ('default', <class 'Warning'>, 4)
E       ('module', <class 'DeprecationWarning'>, 4)

src/zope/testrunner/tests/test_runner.py:207: AssertionError
________________________________________________________________________________ test_suite ________________________________________________________________________________

    def test_suite():
        suite = unittest.TestSuite()
>       suite.addTest(None)

src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_none_test.py:22:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <unittest.suite.TestSuite tests=[]>, test = None

    def addTest(self, test):
        # sanity checks
        if not callable(test):
>           raise TypeError("{} is not callable".format(repr(test)))
E           TypeError: None is not callable

/usr/lib64/python3.8/unittest/suite.py:47: TypeError
========================================================================= short test summary info ==========================================================================
ERROR src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py::test_y0
ERROR src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py::test_x0
ERROR src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py::test_z1
ERROR src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py::test_y0
ERROR src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py::test_x0
ERROR src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py::test_z1
ERROR src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py::test_y0
ERROR src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py::test_x0
ERROR src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py::test_z1
FAILED src/zope/testrunner/tests/test_runner.py::TestWarnings::test_warning_filter_default - AssertionError: Items in the second set but not the first:
FAILED src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_none_test.py::test_suite - TypeError: None is not callable
================================================================== 2 failed, 74 passed, 9 errors in 0.82s ==================================================================

Cannot find what provides self fixture 🤔

d-maurer commented 2 years ago

Tomasz KÅ‚oczko wrote at 2022-6-25 16:02 -0700:

I know that you don't like pytest however I see really good value in use pyterstbecause it is way more actively developed and flexible so ..

We had this discussion already some months ago: a huge number of zopefoundation tests make use of the layer feature: tests are assigned to hierarchically organized layers; the tests on a layer share an test infrastructure, set up by the layer's setUp and torn down by the layer's tearDown. If pytest does not support such a feature, it is inadequate to run the (more precisely most) zopefoundation tests.

kloczek commented 2 years ago

And as you told me as well zope.testrunner was developed at the time when it was no pytest knowing fact that pyets is now used by +90% actively developed pypi modules sooner opr later IMO it will be harder and harder ignore what pytest offer. zope.testrunner looks like effectively is using unittest which is effectively frozen about development, and pytest can handle correctly written unittest` test suite.

Coud you please focus and reply on proposed changes?

d-maurer commented 2 years ago

Tomasz KÅ‚oczko wrote at 2022-6-26 00:42 -0700:

... Coud you please focus and reply on proposed changes?

I will ignore henceforth all your pytest related issues.

icemac commented 2 years ago

@kloczek Please stop spamming zopefoundation repositories with pytest issues. It cannot be used here as your tests show. It is no drop-in replacement for zope.testrunner.