2.0.0: pytest is failing in 3 units

kloczek commented 1 year ago

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

python3 -sBm build -w --no-isolation
because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
install .whl file in </install/prefix> using 'installer` module
run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-crochet-2.0.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-crochet-2.0.0-2.fc35.x86_64/usr/lib/python3.8/site-packages + /usr/bin/pytest -ra -m 'not network' ==================================================================================== test session starts ==================================================================================== platform linux -- Python 3.8.17, pytest-7.4.0, pluggy-1.0.0 rootdir: /home/tkloczko/rpmbuild/BUILD/crochet-2.0.0 collected 114 items crochet/tests/test_api.py F...............s...........................................F... [ 56%] crochet/tests/test_logging.py ..F. [ 59%] crochet/tests/test_mypy.py ssssssssssssss [ 71%] crochet/tests/test_process.py . [ 72%] crochet/tests/test_resultstore.py ..... [ 77%] crochet/tests/test_setup.py ............s...s [ 92%] crochet/tests/test_shutdown.py ..... [ 96%] crochet/tests/test_util.py .... [100%] ========================================================================================= FAILURES ========================================================================================== __________________________________________________________________________ ResultRegistryTests.test_runs_with_lock __________________________________________________________________________ 'NoneType' object is not iterable During handling of the above exception, another exception occurred: NOTE: Incompatible Exception Representation, displaying natively: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean. DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug) (reapAllProcesses, *(), **{})()> _________________________________________________________________________ PublicAPITests.test_eventloop_api_reactor _________________________________________________________________________ self = def test_eventloop_api_reactor(self): """ The publicly exposed EventLoop will, when setup, use the global reactor. """ from twisted.internet import reactor > _main.no_setup() crochet/tests/test_api.py:1118: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ crochet/_util.py:12: in _synced return method(*args, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = @synchronized def no_setup(self): """ Initialize the crochet library with no side effects. No reactor will be started, logging is uneffected, etc.. Future calls to setup() will have no effect. This is useful for applications that intend to run Twisted's reactor themselves, and so do not want libraries using crochet to attempt to start it on their own. If no_setup() is called after setup(), a RuntimeError is raised. """ if self._started: > raise RuntimeError( "no_setup() is intended to be called once, by a" " Twisted application, before any libraries " "using crochet are imported and call setup().") E RuntimeError: no_setup() is intended to be called once, by a Twisted application, before any libraries using crochet are imported and call setup(). crochet/_eventloop.py:393: RuntimeError _______________________________________________________________________ ThreadLogObserverTest.test_ioThreadUnchanged ________________________________________________________________________ self = def test_ioThreadUnchanged(self): """ ThreadLogObserver does not change the Twisted I/O thread (which is supposed to match the thread the main reactor is running in.) """ threadLog = ThreadLogObserver(None) threadLog.stop() threadLog._thread.join() > self.assertIn( threadable.ioThread, # Either reactor was never run, or run in thread running # the tests: (None, threading.current_thread().ident)) crochet/tests/test_logging.py:86: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , containee = 140350667507392, container = (None, 140350712317760), msg = None def assertIn(self, containee, container, msg=None): """ Fail the test if C{containee} is not found in C{container}. @param containee: the value that should be in C{container} @param container: a sequence type, or in the case of a mapping type, will follow semantics of 'if key in dict.keys()' @param msg: if msg is None, then the failure message will be '%r not in %r' % (first, second) """ if containee not in container: > raise self.failureException(msg or f"{containee!r} not in {container!r}") E twisted.trial.unittest.FailTest: 140350667507392 not in (None, 140350712317760) /usr/lib/python3.8/site-packages/twisted/trial/_synctest.py:506: FailTest ===================================================================================== warnings summary ====================================================================================== crochet/tests/test_api.py:15 /home/tkloczko/rpmbuild/BUILD/crochet-2.0.0/crochet/tests/test_api.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp crochet/tests/test_setup.py::ProcessSetupTests::test_posix /home/tkloczko/rpmbuild/BUILD/crochet-2.0.0/crochet/tests/test_setup.py:261: DeprecationWarning: Please use assertEqual instead. self.assertEquals(reaps, [1]) crochet/tests/test_setup.py::ProcessSetupTests::test_posix /home/tkloczko/rpmbuild/BUILD/crochet-2.0.0/crochet/tests/test_setup.py:263: DeprecationWarning: Please use assertEqual instead. self.assertEquals(reaps, [1, 1]) crochet/tests/test_setup.py::ProcessSetupTests::test_posix /home/tkloczko/rpmbuild/BUILD/crochet-2.0.0/crochet/tests/test_setup.py:265: DeprecationWarning: Please use assertEqual instead. self.assertEquals(reaps, [1, 1, 1]) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================================== short test summary info ================================================================================== SKIPPED [1] crochet/tests/test_api.py:497: This test is too fragile (and insufficient) on Python 3 - see https://github.com/itamarst/crochet/issues/43 SKIPPED [1] crochet/tests/test_mypy.py:280: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:337: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:320: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:295: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:366: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:24: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:381: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:140: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:246: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:54: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:31: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:466: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:433: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_mypy.py:404: Tests require mypy to be installed. SKIPPED [1] crochet/tests/test_setup.py:267: This test is for non-POSIX systems. SKIPPED [1] crochet/tests/test_setup.py:329: This test is for Twisted < 15.2. FAILED crochet/tests/test_api.py::ResultRegistryTests::test_runs_with_lock - Failed: NOTE: Incompatible Exception Representation, displaying natively: FAILED crochet/tests/test_api.py::PublicAPITests::test_eventloop_api_reactor - RuntimeError: no_setup() is intended to be called once, by a Twisted application, before any libraries using crochet are imported and call setup(). FAILED crochet/tests/test_logging.py::ThreadLogObserverTest::test_ioThreadUnchanged - twisted.trial.unittest.FailTest: 140350667507392 not in (None, 140350712317760) =================================================================== 3 failed, 94 passed, 17 skipped, 4 warnings in 7.79s ==================================================================== ```

Here is list of installed modules in build env

```console Package Version ----------------------------- ------- alabaster 0.7.13 asttokens 2.2.1 attrs 23.1.0 Automat 22.10.0 Babel 2.12.1 backcall 0.2.0 build 0.10.0 charset-normalizer 3.1.0 constantly 15.1.0 decorator 5.1.1 distro 1.8.0 docutils 0.19 exceptiongroup 1.1.1 executing 1.2.0 gpg 1.20.0 hyperlink 21.0.0 idna 3.4 imagesize 1.4.1 importlib-metadata 6.7.0 incremental 22.10.0 iniconfig 2.0.0 installer 0.7.0 ipython 8.12.0 jedi 0.18.2 Jinja2 3.1.2 libcomps 0.1.19 MarkupSafe 2.1.2 matplotlib-inline 0.1.6 packaging 23.1 parso 0.8.3 pexpect 4.8.0 pickleshare 0.7.5 pluggy 1.0.0 prompt-toolkit 3.0.38 ptyprocess 0.7.0 pure-eval 0.2.2 Pygments 2.15.1 pyproject_hooks 1.0.0 pytest 7.4.0 python-dateutil 2.8.2 pytz 2023.2 requests 2.31.0 setuptools 68.0.0 six 1.16.0 snowballstemmer 2.2.0 Sphinx 6.2.1 sphinxcontrib-applehelp 1.0.4 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 stack-data 0.6.2 tomli 2.0.1 traitlets 5.9.0 Twisted 22.10.0 typing_extensions 4.6.3 urllib3 1.26.15 wcwidth 0.2.6 wheel 0.40.0 wrapt 1.14.1 zipp 3.15.0 zope.event 4.5.0 zope.interface 5.5.2 ```

itamarst commented 1 year ago

The way to run tests is by running tox, e.g. tox -e py311. Or if you want to do it directly, python -m unittest discover -v crochet.tests.

I will however take a look since this suggests fragile tests.

itamarst commented 1 year ago

I found and fixed the issue, but in general I don't recommend just using pytest for any random project; it does Magic™ that occasionally breaks things, as in this case. And I still recommend you use the above command to run Crochet's tests.

kloczek commented 1 year ago

I found and fixed the issue, but in general I don't recommend just using pytest for any random project; it does Magic™ that occasionally breaks things, as in this case. And I still recommend you use the above command to run Crochet's tests.

I'm building packages with python modules in dedicated build envs in which are installed only resources listed in rpm spec file BuildRequirea. In that scenario use tox is kind of double wrapping tox additionally if something is missing downloads .whl archives from pypi (I'm not interested to test anything against pypi but against my own packaged modules.

Just tested new 2.1.1 and all looks good now

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-crochet-2.1.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-crochet-2.1.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.17, pytest-7.4.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/crochet-2.1.1
plugins: mock-3.11.1, anyio-3.6.2, asyncio-0.21.0, aiohttp-1.0.4, httpbin-2.0.0
asyncio: mode=strict
collected 113 items

crochet/tests/test_api.py ................s..............................................                                                                                             [ 55%]
crochet/tests/test_logging.py ....                                                                                                                                                    [ 59%]
crochet/tests/test_mypy.py ..............                                                                                                                                             [ 71%]
crochet/tests/test_process.py .                                                                                                                                                       [ 72%]
crochet/tests/test_resultstore.py .....                                                                                                                                               [ 76%]
crochet/tests/test_setup.py ............s...s                                                                                                                                         [ 92%]
crochet/tests/test_shutdown.py .....                                                                                                                                                  [ 96%]
crochet/tests/test_util.py ....                                                                                                                                                       [100%]

================================================================================== short test summary info ==================================================================================
SKIPPED [1] crochet/tests/test_api.py:497: This test is too fragile (and insufficient) on Python 3 - see https://github.com/itamarst/crochet/issues/43
SKIPPED [1] crochet/tests/test_setup.py:267: This test is for non-POSIX systems.
SKIPPED [1] crochet/tests/test_setup.py:329: This test is for Twisted < 15.2.
========================================================================= 110 passed, 3 skipped in 67.77s (0:01:07) =========================================================================

Thank you 👍 😄

itamarst / crochet

2.0.0: pytest is failing in 3 units #146