1.10.1: pytest is failing and `PytestCollectionWarning` warnings

[kloczek]

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-manuel-1.10.1-14.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-manuel-1.10.1-14.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra src/manuel/testcase.py src/manuel/testing.py src/manuel/tests.py
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/manuel-1.10.1
collected 2 items

src/manuel/testing.py E                                                                                                                                              [ 50%]
src/manuel/tests.py .                                                                                                                                                [100%]

================================================================================== ERRORS ==================================================================================
____________________________________________________________________ ERROR at setup of TestCase.runTest ____________________________________________________________________

cls = <class '_pytest.runner.CallInfo'>, func = <function call_runtest_hook.<locals>.<lambda> at 0x7ffa87cadf70>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

/usr/lib/python3.8/site-packages/_pytest/runner.py:311:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/_pytest/runner.py:255: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/lib/python3.8/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/lib/python3.8/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/lib/python3.8/site-packages/_pytest/runner.py:150: in pytest_runtest_setup
    item.session._setupstate.prepare(item)
/usr/lib/python3.8/site-packages/_pytest/runner.py:452: in prepare
    raise e
/usr/lib/python3.8/site-packages/_pytest/runner.py:449: in prepare
    col.setup()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <TestCaseFunction runTest>

    def setup(self) -> None:
        # A bound method to be called during teardown() if set (see 'runtest()').
        self._explicit_tearDown: Optional[Callable[[], None]] = None
        assert self.parent is not None
>       self._testcase = self.parent.obj(self.name)  # type: ignore[attr-defined]
E       TypeError: __init__() missing 2 required positional arguments: 'regions' and 'globs'

/usr/lib/python3.8/site-packages/_pytest/unittest.py:194: TypeError
============================================================================= warnings summary =============================================================================
src/manuel/testcase.py:9
  /home/tkloczko/rpmbuild/BUILD/manuel-1.10.1/src/manuel/testcase.py:9: DeprecationWarning: invalid escape sequence \s
    SECTION_UNDERLINE = re.compile('^[' + punctuation + ']+\s*$', re.MULTILINE)

src/manuel/testing.py:18
  /home/tkloczko/rpmbuild/BUILD/manuel-1.10.1/src/manuel/testing.py:18: PytestCollectionWarning: cannot collect test class 'TestCaseMarker' because it has a __init__ constructor (from: src/manuel/testing.py)
    class TestCaseMarker(object):

src/manuel/testing.py:199
  /home/tkloczko/rpmbuild/BUILD/manuel-1.10.1/src/manuel/testing.py:199: PytestCollectionWarning: cannot collect test class 'TestFactory' because it has a __init__ constructor (from: src/manuel/testing.py)
    class TestFactory:

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================= short test summary info ==========================================================================
ERROR src/manuel/testing.py::TestCase::runTest - TypeError: __init__() missing 2 required positional arguments: 'regions' and 'globs'
================================================================== 1 passed, 3 warnings, 1 error in 0.34s ==================================================================

[benji-york]

Thanks for the report Tomasz.

There are at least two issues effecting you. One is that—as far as I am aware—the Manuel tests do not work correctly when running under the pytest test runner. In particular, it ignores test_suite function in test files, which is the primary way Manuel tests are discovered.

Secondly, you are instructing pytest to run non-test files. The files src/manuel/testing.py and src/manuel/testcase.py include code to support testing; they do not contain any actual tests.

If it helps, here are a couple of ways to run Manuel's tests:

python -m venv ve
ve/bin/pip install .
ve/bin/python setup.py test

Alternatively, using tox:

python -m venv ve
ve/bin/pip install tox
tox

If there is anything else I can do for you, just let me know.

[kloczek]

As I wrote in the ticket I'm building at the end module to form rpm package. In that case package is build in env of the modules with which it will be used (only those listed in package BuildRequires + dependencies). Using tox adds another wrapping and does not guarantee that test suite will be tested in env of those modules which are used in build env and tox is able to download .whl archive from public network.

In my case using pytest has second goal. I want time to time be able to build package and execute pytest with additionally installed some pytest extensions like pytest-randomly to be able automatically evaluate quality of exact module. Tox always performs test in env of fixed set of modules.

Module editable mode .. that part is already done by build module by default.

Secondly, you are instructing pytest to run non-test files. The files src/manuel/testing.py and src/manuel/testcase.py include code to support testing; they do not contain any actual tests.

AFAIK in pytest.ini is possible to specify such details like list of those files which only shaped be used by pytest when pytest is executed without additional params. If you know that some exact files should not be scanned by pytest the best would be alter that list which is possible to specify in pytest.ini. I can change my spec file to specify that list as param but I think that it would be better if you would have full control of that list between versions 😄

[benji-york]

I'm afraid I can't help you with pytest. Manuel uses a test discovery mechanism that they have explicitly decided not to support (see https://github.com/pytest-dev/pytest/issues/1877). Therefore, it is not currently possible to run the Manuel tests with the pytest runner.

As for avoiding tox, here is an alternate way to run the Manuel tests that does not require tox:

python -m unittest manuel.tests.test_suite

[jamesjer]

@kloczek , if you want to see an example of an RPM spec file for manuel, take a look here.

[kloczek]

Thx but I have mine way simpler 😄

# BUG: pytest is failing https://github.com/benji-york/manuel/issues/22
%bcond_with     failing_tests   # By default skip some failing test units

Summary:        Build tested documentation
Name:           python-manuel
Version:        1.10.1
Release:        14%{?dist}
License:        ASL-2.0 (https://www.apache.org/licenses/LICENSE-2.0)
URL:            https://pypi.python.org/pypi/manuel/
VCS:            https://github.com/benji-york/manuel/
Source0:        %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
BuildArch:      noarch
BuildRequires:  python3dist(build)
BuildRequires:  python3dist(pip)
BuildRequires:  python3dist(setuptools)
BuildRequires:  python3dist(docutils)
BuildRequires:  python3dist(six)
BuildRequires:  python3dist(zope-testing)
BuildRequires:  python3dist(wheel)
# CheckRequires:
BuildRequires:  python3dist(pytest)
Obsoletes:      python3-manuel

%description
Manuel lets you mix and match traditional doctests with custom test syntax.
Several plug-ins are included that provide new test syntax. You can also create
your own plug-ins.

%prep
%autosetup -n manuel-%{version}

%build
%pyproject_wheel

%install
%pyproject_install

%check
%pytest src/manuel/test*.py %{!?with_failing_tests: \
        --deselect src/manuel/testing.py::TestCase::runTest \
}

%files
%{python3_sitelib}/manuel
%{python3_sitelib}/manuel-*.*-info

Generally speqking I'm interested to test all possible module using only pytest because it provides possibility to test all modules by adding to build env pytest extensions and without touching spec file😄

[tkloczko@devel-g2v SPECS]$ ls python-* | wc -l; grep %pytest python-* |wc -l; grep %tox python-* |wc -l
877
774
2

[kloczek]

Looks like 1.11.2 still cannot be used with pytest 😞

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-manuel-1.11.2-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-manuel-1.11.2-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra src/manuel/testcase.py src/manuel/testing.py src/manuel/tests.py
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/manuel-1.11.2
collected 2 items

src/manuel/testing.py E                                                                                                                                              [ 50%]
src/manuel/tests.py .                                                                                                                                                [100%]

================================================================================== ERRORS ==================================================================================
____________________________________________________________________ ERROR at setup of TestCase.runTest ____________________________________________________________________

cls = <class '_pytest.runner.CallInfo'>, func = <function call_runtest_hook.<locals>.<lambda> at 0x7f9db01ae430>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.

        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

/usr/lib/python3.8/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/_pytest/runner.py:259: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/lib/python3.8/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/lib/python3.8/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/lib/python3.8/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
    item.session._setupstate.setup(item)
/usr/lib/python3.8/site-packages/_pytest/runner.py:494: in setup
    raise exc
/usr/lib/python3.8/site-packages/_pytest/runner.py:491: in setup
    col.setup()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <TestCaseFunction runTest>

    def setup(self) -> None:
        # A bound method to be called during teardown() if set (see 'runtest()').
        self._explicit_tearDown: Optional[Callable[[], None]] = None
        assert self.parent is not None
>       self._testcase = self.parent.obj(self.name)  # type: ignore[attr-defined]
E       TypeError: __init__() missing 2 required positional arguments: 'regions' and 'globs'

/usr/lib/python3.8/site-packages/_pytest/unittest.py:201: TypeError
============================================================================= warnings summary =============================================================================
src/manuel/testcase.py:9
  /home/tkloczko/rpmbuild/BUILD/manuel-1.11.2/src/manuel/testcase.py:9: DeprecationWarning: invalid escape sequence \s
    SECTION_UNDERLINE = re.compile('^[' + punctuation + ']+\s*$', re.MULTILINE)

src/manuel/testing.py:18
  /home/tkloczko/rpmbuild/BUILD/manuel-1.11.2/src/manuel/testing.py:18: PytestCollectionWarning: cannot collect test class 'TestCaseMarker' because it has a __init__ constructor (from: src/manuel/testing.py)
    class TestCaseMarker(object):

src/manuel/testing.py:203
  /home/tkloczko/rpmbuild/BUILD/manuel-1.11.2/src/manuel/testing.py:203: PytestCollectionWarning: cannot collect test class 'TestFactory' because it has a __init__ constructor (from: src/manuel/testing.py)
    class TestFactory:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================= short test summary info ==========================================================================
ERROR src/manuel/testing.py::TestCase::runTest - TypeError: __init__() missing 2 required positional arguments: 'regions' and 'globs'
================================================================== 1 passed, 3 warnings, 1 error in 0.22s ==================================================================

However after deselecting that one unit seems is OK .. and there are some warnings.

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-manuel-1.11.2-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-manuel-1.11.2-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra src/manuel/testcase.py src/manuel/testing.py src/manuel/tests.py --deselect src/manuel/testing.py::TestCase::runTest
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/manuel-1.11.2
collected 2 items / 1 deselected / 1 selected

src/manuel/tests.py .                                                                                                                                                [100%]

============================================================================= warnings summary =============================================================================
src/manuel/testcase.py:9
  /home/tkloczko/rpmbuild/BUILD/manuel-1.11.2/src/manuel/testcase.py:9: DeprecationWarning: invalid escape sequence \s
    SECTION_UNDERLINE = re.compile('^[' + punctuation + ']+\s*$', re.MULTILINE)

src/manuel/testing.py:18
  /home/tkloczko/rpmbuild/BUILD/manuel-1.11.2/src/manuel/testing.py:18: PytestCollectionWarning: cannot collect test class 'TestCaseMarker' because it has a __init__ constructor (from: src/manuel/testing.py)
    class TestCaseMarker(object):

src/manuel/testing.py:203
  /home/tkloczko/rpmbuild/BUILD/manuel-1.11.2/src/manuel/testing.py:203: PytestCollectionWarning: cannot collect test class 'TestFactory' because it has a __init__ constructor (from: src/manuel/testing.py)
    class TestFactory:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================================================== 1 passed, 1 deselected, 3 warnings in 0.10s ================================================================

[kloczek]

Gentle ping .. 😃

[benji-york]

I have no current plans to work on pytest support for Manuel.