plugins missing running tests in RPM build environment

kloczek commented 2 years ago

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

python3 -sBm build -w --no-isolation
because I'm calling build with --no-isolation I'm using during all processes oly locally installed modules
install .whl file in </install/prefix>
run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-tempora-4.1.2-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-tempora-4.1.2-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -p no:flake8
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/tempora-4.1.2, configfile: pytest.ini
collected 45 items

tempora/__init__.py ..........F....                                                                                                                                  [ 33%]
tempora/schedule.py ...                                                                                                                                              [ 40%]
tempora/timing.py .....                                                                                                                                              [ 51%]
tempora/utc.py .                                                                                                                                                     [ 53%]
tempora/tests/test_schedule.py ...................                                                                                                                   [ 95%]
tempora/tests/test_timing.py ..                                                                                                                                      [100%]

================================================================================= FAILURES =================================================================================
________________________________________________________________ [doctest] tempora.get_nearest_year_for_day ________________________________________________________________
269
270     Returns the nearest year to now inferred from a Julian date.
271
272     >>> freezer = getfixture('freezer')
UNEXPECTED EXCEPTION: FixtureLookupError('freezer', <FixtureRequest for <DoctestItem tempora.get_nearest_year_for_day>>)
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/_pytest/fixtures.py", line 589, in _get_active_fixturedef
    return self._fixture_defs[argname]
KeyError: 'freezer'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib64/python3.8/doctest.py", line 1336, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest tempora.get_nearest_year_for_day[0]>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/_pytest/fixtures.py", line 581, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/usr/lib/python3.8/site-packages/_pytest/fixtures.py", line 592, in _get_active_fixturedef
    fixturedef = self._getnextfixturedef(argname)
  File "/usr/lib/python3.8/site-packages/_pytest/fixtures.py", line 473, in _getnextfixturedef
    raise FixtureLookupError(argname, self)
_pytest.fixtures.FixtureLookupError: ('freezer', <FixtureRequest for <DoctestItem tempora.get_nearest_year_for_day>>)
/home/tkloczko/rpmbuild/BUILD/tempora-4.1.2/tempora/__init__.py:272: UnexpectedException
========================================================================= short test summary info ==========================================================================
FAILED tempora/__init__.py::tempora.get_nearest_year_for_day
======================================================================= 1 failed, 44 passed in 1.55s =======================================================================

List of modules installed in build env:

Package                       Version
----------------------------- -------------------
alabaster                     0.7.12
appdirs                       1.4.4
attrs                         21.4.0
Babel                         2.9.1
build                         0.7.0
chardet                       4.0.0
charset-normalizer            2.0.10
cycler                        0.11.0
dbus-python                   1.2.18
distro                        1.6.0
docutils                      0.17.1
extras                        1.0.0
fixtures                      3.0.0
fonttools                     4.28.5
freezegun                     1.0.0
gpg                           1.16.0-unknown
idna                          3.3
imagesize                     1.3.0
importlib-metadata            4.10.0
iniconfig                     1.1.1
jaraco.functools              3.5.0
jaraco.packaging              8.2.1
Jinja2                        3.0.3
kiwisolver                    1.3.2
libcomps                      0.1.18
lit                           13.0.0
lxml                          4.7.1
Mako                          1.1.6.dev0
Markdown                      3.3.6
MarkupSafe                    2.0.1
matplotlib                    3.5.1
meson                         0.60.3
more-itertools                8.12.0
numpy                         1.22.0
olefile                       0.46
packaging                     21.3
pbr                           5.8.0
pep517                        0.12.0
Pillow                        9.0.0
pip                           21.3.1
pluggy                        1.0.0
py                            1.11.0
Pygments                      2.11.2
PyGObject                     3.42.0
pyparsing                     3.0.6
pytest                        6.2.5
python-dateutil               2.8.2
pytz                          2021.1
requests                      2.27.1
rpm                           4.17.0
rst.linker                    2.2.0
scour                         0.38.2
setuptools                    60.1.0.post20211225
setuptools-scm                6.3.2
six                           1.16.0
smartypants                   2.0.1
snowballstemmer               2.2.0
Sphinx                        4.3.2.dev20220106
sphinxcontrib-applehelp       1.0.2.dev20211227
sphinxcontrib-devhelp         1.0.2.dev20211228
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1.dev20211227
sphinxcontrib-qthelp          1.0.3.dev20211227
sphinxcontrib-serializinghtml 1.1.5
testtools                     2.5.0
toml                          0.10.2
tomli                         2.0.0
typogrify                     2.0.7
urllib3                       1.26.7
wheel                         0.37.1
zipp                          3.7.0

jaraco commented 2 years ago

I get the same error if I apply this diff to the project and run tests with tox:

tempora main $ git diff
diff --git a/setup.cfg b/setup.cfg
index d29a911..7c70a7c 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -45,7 +45,6 @@ testing =
        # local
        backports.unittest_mock
        freezegun
-       pytest-freezegun
        types-freezegun
        types-pytz

I also get the error if I run the tests with tox -- -p no:freezegun.

That leads me to believe that for whatever reason, your build environment isn't including all of the test dependencies (pytest-freezegun in particular) or somehow the metadata from those packages is missing.

I notice in your pytest output, none of the plugins are loaded. I expect to see something like:

============================================================= test session starts =============================================================
platform darwin -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
cachedir: .tox/python/.pytest_cache
rootdir: /Users/jaraco/code/main/tempora, configfile: pytest.ini
plugins: flake8-1.0.7, enabler-1.2.1, backports.unittest-mock-1.5, freezegun-0.4.2, checkdocs-2.7.1, black-0.3.12, mypy-0.8.1, cov-3.0.0

Since there are no plugins in your output, my suspicion is that either the test requirements haven't been installed or they've not been installed with the necessary metadata. I'd expect you to see something like this:

python -c "import importlib.metadata as md; import pprint; pprint.pprint(md.entry_points()['pytest11'])"
<string>:1: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
[EntryPoint(name='flake8', value='pytest_flake8', group='pytest11'),
 EntryPoint(name='plugin-enabled options', value='pytest_enabler', group='pytest11'),
 EntryPoint(name='unittest_mock', value='backports.unittest_mock', group='pytest11'),
 EntryPoint(name='freezegun', value='pytest_freezegun', group='pytest11'),
 EntryPoint(name='checkdocs', value='pytest_checkdocs', group='pytest11'),
 EntryPoint(name='black', value='pytest_black', group='pytest11'),
 EntryPoint(name='mypy', value='pytest_mypy', group='pytest11'),
 EntryPoint(name='pytest_cov', value='pytest_cov.plugin', group='pytest11')]

If the packages aren't installed, you'll want to install them. If they are installed, you'll want to investigate why their associated "dist-info" directories are missing.

kloczek commented 2 years ago

Just tested 5.0.0 and all looks good

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-tempora-5.0.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-tempora-5.0.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -p no:flake8 --deselect tempora/__init__.py::tempora.get_nearest_year_for_day
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/tempora-5.0.0, configfile: pytest.ini
collected 42 items / 1 deselected / 41 selected

tempora/__init__.py ...........                                                                                                                                      [ 26%]
tempora/schedule.py ...                                                                                                                                              [ 34%]
tempora/timing.py .....                                                                                                                                              [ 46%]
tempora/utc.py .                                                                                                                                                     [ 48%]
tempora/tests/test_schedule.py ...................                                                                                                                   [ 95%]
tempora/tests/test_timing.py ..                                                                                                                                      [100%]

===================================================================== 41 passed, 1 deselected in 1.53s =====================================================================

kloczek commented 2 years ago

Nope .. sorry. Forgot do deselect previously failing unit :/

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-tempora-5.0.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-tempora-5.0.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/tempora-5.0.0, configfile: pytest.ini
plugins: flake8-1.0.7
collected 42 items

tempora/__init__.py .......F....                                                                                                                                     [ 28%]
tempora/schedule.py ...                                                                                                                                              [ 35%]
tempora/timing.py .....                                                                                                                                              [ 47%]
tempora/utc.py .                                                                                                                                                     [ 50%]
tempora/tests/test_schedule.py ...................                                                                                                                   [ 95%]
tempora/tests/test_timing.py ..                                                                                                                                      [100%]

================================================================================= FAILURES =================================================================================
________________________________________________________________ [doctest] tempora.get_nearest_year_for_day ________________________________________________________________
203
204     Returns the nearest year to now inferred from a Julian date.
205
206     >>> freezer = getfixture('freezer')
UNEXPECTED EXCEPTION: FixtureLookupError('freezer', <FixtureRequest for <DoctestItem tempora.get_nearest_year_for_day>>)
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/_pytest/fixtures.py", line 589, in _get_active_fixturedef
    return self._fixture_defs[argname]
KeyError: 'freezer'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib64/python3.8/doctest.py", line 1336, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest tempora.get_nearest_year_for_day[0]>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/_pytest/fixtures.py", line 581, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/usr/lib/python3.8/site-packages/_pytest/fixtures.py", line 592, in _get_active_fixturedef
    fixturedef = self._getnextfixturedef(argname)
  File "/usr/lib/python3.8/site-packages/_pytest/fixtures.py", line 473, in _getnextfixturedef
    raise FixtureLookupError(argname, self)
_pytest.fixtures.FixtureLookupError: ('freezer', <FixtureRequest for <DoctestItem tempora.get_nearest_year_for_day>>)
/home/tkloczko/rpmbuild/BUILD/tempora-5.0.0/tempora/__init__.py:206: UnexpectedException
========================================================================= short test summary info ==========================================================================
FAILED tempora/__init__.py::tempora.get_nearest_year_for_day
======================================================================= 1 failed, 41 passed in 1.53s =======================================================================

jaraco commented 2 years ago

Please see my analysis - the issue remains the same - a test dependency pytest-freezegun is missing.

kloczek commented 2 years ago

Error message was misleading. With installed pytest-frezgun pytest is OK 😄

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-tempora-5.0.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-tempora-5.0.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/tempora-5.0.1, configfile: pytest.ini
plugins: flaky-3.7.0, forked-1.4.0, shutil-1.7.0, cov-3.0.0, virtualenv-1.7.0, flake8-1.0.7, xdist-2.5.0, checkdocs-2.7.1, freezegun-0.4.2
collected 45 items

. .                                                                                                                                                                  [  2%]
tempora/__init__.py ............                                                                                                                                     [ 30%]
tempora/schedule.py ...                                                                                                                                              [ 37%]
tempora/timing.py .....                                                                                                                                              [ 48%]
tempora/utc.py .                                                                                                                                                     [ 51%]
tempora/tests/test_schedule.py ...................                                                                                                                   [ 95%]
tempora/tests/test_timing.py ..                                                                                                                                      [100%]

============================================================================= warnings summary =============================================================================
../../../../../usr/lib/python3.8/site-packages/pytest_freezegun.py:17: 86 warnings
tempora/__init__.py: 2 warnings
  /usr/lib/python3.8/site-packages/pytest_freezegun.py:17: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if LooseVersion(pytest.__version__) < LooseVersion('3.6.0'):

-- Docs: https://docs.pytest.org/en/stable/warnings.html
===================================================================== 43 passed, 88 warnings in 8.25s ======================================================================

Feel free to close this tisket if you are OK about above pytet warning.

Thank you.

kloczek commented 2 years ago

Closing. Gosh .. that warning is in pytest-freezegun code 😄

jaraco / tempora

plugins missing running tests in RPM build environment #20