Closed kloczek closed 2 years ago
Here is pytest output:
Use zope.testrunner
to run the tests.
As at the monebt pytest is de facto standard pytest should be supported as well. Just retested 2.4 and pytest is failing a bit different now
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-zodbpickle-2.4-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-zodbpickle-2.4-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.13, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/zodbpickle-2.4
collected 8 items
src/zodbpickle/tests/test_compile_flags.py . [ 12%]
src/zodbpickle/tests/test_pickle.py ..FFFF. [100%]
================================================================================= FAILURES =================================================================================
_______________________________________________________________________ TestImportability.test_dump ________________________________________________________________________
self = <zodbpickle.tests.test_pickle.TestImportability testMethod=test_dump>
def test_dump(self):
from zodbpickle.pickle import dump
> self.assertIsInstance(dump, function_type)
E AssertionError: <function _dump at 0x7f9b5ea3e550> is not an instance of <class 'builtin_function_or_method'>
src/zodbpickle/tests/test_pickle.py:36: AssertionError
_______________________________________________________________________ TestImportability.test_dumps _______________________________________________________________________
self = <zodbpickle.tests.test_pickle.TestImportability testMethod=test_dumps>
def test_dumps(self):
from zodbpickle.pickle import dumps
> self.assertIsInstance(dumps, function_type)
E AssertionError: <function _dumps at 0x7f9b5ea3e5e0> is not an instance of <class 'builtin_function_or_method'>
src/zodbpickle/tests/test_pickle.py:40: AssertionError
_______________________________________________________________________ TestImportability.test_load ________________________________________________________________________
self = <zodbpickle.tests.test_pickle.TestImportability testMethod=test_load>
def test_load(self):
from zodbpickle.pickle import load
> self.assertIsInstance(load, function_type)
E AssertionError: <function _load at 0x7f9b5ea3e670> is not an instance of <class 'builtin_function_or_method'>
src/zodbpickle/tests/test_pickle.py:28: AssertionError
_______________________________________________________________________ TestImportability.test_loads _______________________________________________________________________
self = <zodbpickle.tests.test_pickle.TestImportability testMethod=test_loads>
def test_loads(self):
from zodbpickle.pickle import loads
> self.assertIsInstance(loads, function_type)
E AssertionError: <function _loads at 0x7f9b5ea3e700> is not an instance of <class 'builtin_function_or_method'>
src/zodbpickle/tests/test_pickle.py:32: AssertionError
========================================================================= short test summary info ==========================================================================
FAILED src/zodbpickle/tests/test_pickle.py::TestImportability::test_dump - AssertionError: <function _dump at 0x7f9b5ea3e550> is not an instance of <class 'builtin_funct...
FAILED src/zodbpickle/tests/test_pickle.py::TestImportability::test_dumps - AssertionError: <function _dumps at 0x7f9b5ea3e5e0> is not an instance of <class 'builtin_fun...
FAILED src/zodbpickle/tests/test_pickle.py::TestImportability::test_load - AssertionError: <function _load at 0x7f9b5ea3e670> is not an instance of <class 'builtin_funct...
FAILED src/zodbpickle/tests/test_pickle.py::TestImportability::test_loads - AssertionError: <function _loads at 0x7f9b5ea3e700> is not an instance of <class 'builtin_fun...
======================================================================= 4 failed, 4 passed in 0.21s ========================================================================
BTW looks like test suite is installed. Here is the patch to exclude test suite files from what is installed.
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -12,3 +12,4 @@
include *.sh
recursive-include patches *.diff
recursive-include src *.c
+exclude src/zodbpickle/tests
pytest is very popular, yes, but it's not what we use across the entire ZopeFoundation repository, and porting the test suite to work properly with pytest is a large job that nobody has volunteered to do so far.
Marius Gedminas wrote at 2022-9-18 07:52 -0700:
pytest is very popular, yes, but it's not what we use across the entire ZopeFoundation repository, and porting the test suite to work properly with pytest is a large job that nobody has volunteered to do so far.
In addition, many of the suites make important use of the
zope.testrunner
feature (hierarchical) layer
. This allows related tests
to share a common setup (set up once before those tests,
shared by the tests and torn down after them).
It is important because many tests have a very expensive
setup; sharing allows to reduce testing time.
I asked our pytest
protagonist in the past whether pytest
has
a comparable feature -- and have gotten no answer.
Most of the zope modules which I have packaged have no problems with use pytest.
[tkloczko@devel-g2v SPECS]$ ls python*zope*
python-sphinxcontrib-zopeext.spec python-zope-event.spec python-zope-i18nmessageid.spec python-zope-testing.spec
python-zope-component.spec python-zope-exceptions.spec python-zope-interface.spec python-zope-testrunner.spec
python-zope-configuration.spec python-zope-hookable.spec python-zope-schema.spec
[tkloczko@devel-g2v SPECS]$ ls python*zope* |xargs grep %bcond_with.*failing_tests
python-zope-component.spec:%bcond_with failing_tests # By default skip some failing test units
python-zope-configuration.spec:%bcond_with failing_tests # By default skip some failing test units
python-zope-exceptions.spec:%bcond_with failing_tests # By default skip some failing test units
python-zope-testrunner.spec:%bcond_with failing_tests # By default skip some failing test units
[tkloczko@devel-g2v SPECS]$ ls python*zope* |xargs grep %bcond_with.*failing_tests -l | xargs grep BUG
python-zope-component.spec:# BUG: pytest is failing https://github.com/zopefoundation/zope.component/issues/64
python-zope-configuration.spec:# BUG: pytest is failing https://github.com/zopefoundation/zope.configuration/issues/55
python-zope-exceptions.spec:# BUG: pytest is failing https://github.com/zopefoundation/zope.exceptions/issues/21
python-zope-exceptions.spec:# BUG: sphinx warnings https://github.com/zopefoundation/zope.exceptions/issues/22
python-zope-testrunner.spec:# BUG: test suite is failing https://github.com/zopefoundation/zope.testrunner/issues/136
Nevertheless currently testrunner
is way behind of what pytest
can do/provide.
Personnally currently I don't see to much sense to stick to testrunner
as it barrely evolving (but this is only my personall opinion and I fully understand that someone may not agree). testrunner
was born before pytest
and currently it is quite big set of reasons why +95% of all modules are now tesable using pytest
. With some additional modyfications I've reached even higher ratio
[tkloczko@devel-g2v SPECS]$ grep ^%pytest python-*spec -l | wc -l; ls -1 python-*.spec | wc -l
998
1025
But all that is a bit off topic.
In addition, many of the suites make important use of the
zope.testrunner
feature(hierarchical) layer
. This allows related tests to share a common setup (set up once before those tests, shared by the tests and torn down after them). It is important because many tests have a very expensive setup; sharing allows to reduce testing time.
(Note that zodbpickle doesn't use layers, so this is technically offtopic for this issue.)
pytest has session-level fixtures which run setup and teardown at most once during the test session. I'm not sure pytest orders tests by fixture use, so there are any test suites that rely on zope.testrunner layers to clean up any global monkey-patching before running the next test layer, that might not work after porting to pytest. Also, pytest has no direct support for running tests in a subprocess, which zope.testrunner uses to handle layers that raise NotImplementedError in the layer teardown. (You can use a plugin like pytest-xdist to run tests in a different process, but you won't get the automatic grouping by test layer.)
Personnally currently I don't see to much sense to stick to testrunner as it barrely evolving (but this is only my personall opinion and I fully understand that someone may not agree).
As I said, migration takes work. Even if a test suite runs under pytest, have you verified that it runs the same tests? Like here you're showing us a pytest run with 8 tests, but the actual zodbpickle test suite contains 427 tests. The reason most of them weren't picked up is that we're relying on zope.testrunner's feature where it calls test_suite
in each test module to collect the tests, and we use that here to pick the right test file, depending on the major Python version.
Have you verified that the other packages where you chose to use pytest actually run all the tests? We rely on test_suite()
quite often, e.g. to provide custom setUp/tearDown/globs to different doctest files.
It's all solvable, but solving it takes work.
Have you verified that the other packages where you chose to use pytest actually run all the tests? We rely on
test_suite()
quite often, e.g. to provide custom setUp/tearDown/globs to different doctest files.It's all solvable, but solving it takes work.
And that is easy to explain because pyetst only scans for units test_.py files. and in src/zodbpickle/tests/ there are pickle.py so to allow pytest scan for more units some files needs to be renamed.
I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.
python3 -sBm build -w --no-isolation
build
with--no-isolation
I'm using during all processes only locally installed modules