Updating to 36.5.0 breaks tests

stratakis commented 7 years ago

Trying to update the package in Fedora from 36.2.0 to 36.5.0 produces some test failures with the message: RuntimeError: maximum recursion depth exceeded while calling a Python object

Attaching two different build logs where tests are invoked under python2 and under python3 respectively.

py2build.log

py3build.log

jaraco commented 6 years ago

Thanks for the report.

We see six failures on Python 2 and seven on Python 3. Here's the first of the failures seen on the Python 2:

__________________________ test_dist_fetch_build_egg ___________________________
tmpdir = local('/tmp/pytest-of-mockbuild/pytest-0/test_dist_fetch_build_egg0')
    def test_dist_fetch_build_egg(tmpdir):
        """
        Check multiple calls to `Distribution.fetch_build_egg` work as expected.
        """
        index = tmpdir.mkdir('index')
        index_url = urljoin('file://', pathname2url(str(index)))
        def sdist_with_index(distname, version):
            dist_dir = index.mkdir(distname)
            dist_sdist = '%s-%s.tar.gz' % (distname, version)
            make_nspkg_sdist(str(dist_dir.join(dist_sdist)), distname, version)
            with dist_dir.join('index.html').open('w') as fp:
                fp.write(DALS(
                    '''
                    <!DOCTYPE html><html><body>
                    <a href="{dist_sdist}" rel="internal">{dist_sdist}</a><br/>
                    </body></html>
                    '''
                ).format(dist_sdist=dist_sdist))
        sdist_with_index('barbazquux', '3.2.0')
        sdist_with_index('barbazquux-runner', '2.11.1')
        with tmpdir.join('setup.cfg').open('w') as fp:
            fp.write(DALS(
                '''
                [easy_install]
                index_url = {index_url}
                '''
            ).format(index_url=index_url))
        reqs = '''
        barbazquux-runner
        barbazquux
        '''.split()
        with tmpdir.as_cwd():
            dist = Distribution()
            resolved_dists = [
                dist.fetch_build_egg(r)
>               for r in reqs
            ]
setuptools/tests/test_dist.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
setuptools/dist.py:522: in fetch_build_egg
    return cmd.easy_install(req)
setuptools/command/easy_install.py:671: in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
setuptools/command/easy_install.py:697: in install_item
    dists = self.install_eggs(spec, download, tmpdir)
setuptools/command/easy_install.py:878: in install_eggs
    return self.build_and_install(setup_script, setup_base)
setuptools/command/easy_install.py:1117: in build_and_install
    self.run_setup(setup_script, setup_base, args)
setuptools/command/easy_install.py:1103: in run_setup
    run_setup(setup_script, args)
setuptools/sandbox.py:257: in run_setup
    raise
/usr/lib64/python2.7/contextlib.py:35: in __exit__
    self.gen.throw(type, value, traceback)
setuptools/sandbox.py:199: in setup_context
    yield
/usr/lib64/python2.7/contextlib.py:35: in __exit__
    self.gen.throw(type, value, traceback)
setuptools/sandbox.py:170: in save_modules
    saved_exc.resume()
setuptools/sandbox.py:145: in resume
    six.reraise(type, exc, self._tb)
setuptools/sandbox.py:158: in save_modules
    yield saved
setuptools/sandbox.py:199: in setup_context
    yield
E   RuntimeError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)
----------------------------- Captured stdout call -----------------------------
Searching for barbazquux-runner
Reading file:///tmp/pytest-of-mockbuild/pytest-0/test_dist_fetch_build_egg0/index/barbazquux-runner/
Best match: barbazquux-runner 2.11.1
Processing barbazquux-runner-2.11.1.tar.gz
Writing /tmp/easy_install-N4Cer1/setup.cfg
Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-N4Cer1/egg-dist-tmp-sBSZb5
________ TestDistutilsPackage.test_bdist_egg_available_on_distutils_pkg ________
self = <setuptools.tests.test_easy_install.TestDistutilsPackage instance at 0x7f6b1e5e8908>
distutils_package = None
    def test_bdist_egg_available_on_distutils_pkg(self, distutils_package):
>       run_setup('setup.py', ['bdist_egg'])
/builddir/build/BUILD/setuptools-36.5.0/setuptools/tests/test_easy_install.py:323: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/builddir/build/BUILD/setuptools-36.5.0/setuptools/sandbox.py:257: in run_setup
    raise
/usr/lib64/python2.7/contextlib.py:35: in __exit__
    self.gen.throw(type, value, traceback)
/builddir/build/BUILD/setuptools-36.5.0/setuptools/sandbox.py:199: in setup_context
    yield
/usr/lib64/python2.7/contextlib.py:35: in __exit__
    self.gen.throw(type, value, traceback)
/builddir/build/BUILD/setuptools-36.5.0/setuptools/sandbox.py:170: in save_modules
    saved_exc.resume()
/builddir/build/BUILD/setuptools-36.5.0/setuptools/sandbox.py:145: in resume
    six.reraise(type, exc, self._tb)
/builddir/build/BUILD/setuptools-36.5.0/setuptools/sandbox.py:158: in save_modules
    yield saved
/builddir/build/BUILD/setuptools-36.5.0/setuptools/sandbox.py:199: in setup_context
    yield
E   RuntimeError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)

I'm afraid I just don't see what might be going on here.

Can you bisect the changes between 36.2 and 36.5 to see in which release the problem began? If I had to guess, I would think it's 36.4.0.

Can you imagine what it might be about the Fedora environment that elicits this failure?

stratakis commented 6 years ago

Hello and thanks for looking at the issue. I'll try to provide the information you requested.

stratakis commented 6 years ago

Managed to figure out one of the culprits and it is pytest.

Tested so far only on python2 and it seems that since the release of pytest 3.0.0 those four test cases from test_easy_install shown at the build log, started failing with the maximum recursion error.

The same tests actually pass with pytest 2.9.2.

benoit-pierre commented 6 years ago

What version of pytest are you using exactly?

stratakis commented 6 years ago

pytest 3.2.3 currently.

benoit-pierre commented 6 years ago

I can reproduce the issue using a virtualenv if I pin pytest-shutil to 1.2.6: rm -rf venv && python -m venv venv && ./venv/bin/pip install -r tests/requirements.txt 'pytest-shutil==1.2.6' -e . && ./venv/bin/pytest setuptools/tests:

============================= test session starts ==============================
platform linux -- Python 3.6.3, pytest-3.2.3, py-1.4.34, pluggy-0.4.0
rootdir: /home/bpierre/progs/src/setuptools, inifile: pytest.ini
plugins: virtualenv-1.2.11, shutil-1.2.6, flake8-0.9.1
collected 240 items / 2 skipped

setuptools/tests/test_archive_util.py X
setuptools/tests/test_bdist_egg.py .
setuptools/tests/test_build_clib.py .
setuptools/tests/test_build_ext.py ..
setuptools/tests/test_build_py.py .
setuptools/tests/test_config.py .....................
setuptools/tests/test_dep_util.py .
setuptools/tests/test_depends.py .
setuptools/tests/test_develop.py ss....
setuptools/tests/test_dist.py F
setuptools/tests/test_dist_info.py ..
setuptools/tests/test_easy_install.py ......F......FFFF.........
setuptools/tests/test_egg_info.py ...................x..........
setuptools/tests/test_find_packages.py ............
setuptools/tests/test_install_scripts.py .s.s
setuptools/tests/test_integration.py sssssssss
setuptools/tests/test_manifest.py ...................................................................
setuptools/tests/test_namespaces.py ...
setuptools/tests/test_packageindex.py ..................
setuptools/tests/test_sandbox.py .........F
setuptools/tests/test_sdist.py ..........
setuptools/tests/test_setuptools.py ...
setuptools/tests/test_test.py .
setuptools/tests/test_unicode_utils.py .
setuptools/tests/test_upload_docs.py ..
setuptools/tests/test_virtualenv.py ...
setuptools/tests/test_windows_wrappers.py sss

stratakis commented 6 years ago

I actually tried with shutil 1.2.11 and I still get the failures.

benoit-pierre commented 6 years ago

When switching to 1.2.11, the only difference is pytest-shutil itself (same versions for all other packages):

apipkg==1.4
contextlib2==0.5.5
execnet==1.5.0
flake8==3.5.0
mccabe==0.6.1
mock==2.0.0
path.py==10.5
pbr==3.1.1
py==1.4.34
pycodestyle==2.3.1
pyflakes==1.6.0
pytest==3.2.3
pytest-fixture-config==1.2.11
pytest-flake8==0.9.1
pytest-shutil==1.2.11
pytest-virtualenv==1.2.11
six==1.11.0
virtualenv==15.1.0

What versions are you using for those?

stratakis commented 6 years ago

Differences: execnet==1.4.1 path.py==5.2 pytest-shutil==1.2.6

Those packages do not exist in the buildroot, we don't pull them to build setuptools: flake8==3.5.0 mccabe==0.6.1 pycodestyle==2.3.1 pyflakes==1.6.0 pytest-flake8==0.9.1

The rest are the same.

benoit-pierre commented 6 years ago

I get failures with this environment:

apipkg==1.4
contextlib2==0.5.5
execnet==1.4.1
mock==2.0.0
path.py==5.2
pbr==3.1.1
py==1.4.34
pytest==3.2.3
pytest-fixture-config==1.2.11
pytest-shutil==1.2.6
pytest-virtualenv==1.2.11
six==1.11.0
virtualenv==15.1.0

but it works if pytest-shutil is updated to 1.2.11. Note again that this is with Python 3, Python 2 works fine in both cases.

How are all those dependencies provided? From distribution packages? Because another difference could be unvendoring...

stratakis commented 6 years ago

Those are from distribution packages yes, so there might be some deviations from the upstream code in some of those packages but not something that might affect that behaviour, AFAIK.

Also unvendoring of what could possibly cause this? We do not unvendor what setuptools carries within though.

Noting that for the subsequent comments/tests after the initial report, were running on python2, as I'd like to address that first. Will see if I'm able to reproduce it in a virtualenv.

stratakis commented 6 years ago

Also @mcyprian you might wanna take a look at it.

stratakis commented 6 years ago

If however pin pytest to 2.9.2 with rm -rf venv && python -m venv venv && ./venv/bin/pip install -r tests/requirements.txt 'pytest==2.9.2' -e . && ./venv/bin/py.test setuptools/tests and remove as well the requirement for pytest-flake8 due to it requiring pytest>=3.2, the tests pass.

benoit-pierre commented 6 years ago

Don't you get an error with test_egg_info when using pytest==2.9.2:

________________________________________________________________________________ ERROR collecting setuptools/tests/test_egg_info.py ________________________________________________________________________________
setuptools/tests/test_egg_info.py:24: in <module>
    class TestEggInfo(object):
setuptools/tests/test_egg_info.py:368: in TestEggInfo
    mismatch_marker_alternate=mismatch_marker_alternate,
setuptools/tests/test_egg_info.py:216: in parametrize
    argvalues.append(pytest.param(requires, use_cfg,
E   AttributeError: module 'pytest' has no attribute 'param'

?

And using the same command line without pinning pytest, you get the same failures as originally reported?

stratakis commented 6 years ago

@benoit-pierre Yep I get the error that you mention.

If I try however to reproduce the issues outside the build system, and replicate your environment I get the exact same behaviour that you get. Invoking the tests under python 2 is fine, while under python3 I get the recursion error if I pin shutil to 1.2.6.

I think the actual traceback is hidden due to pytest's recursion error. Will see if I can manage to extract a more meaningful traceback.

stratakis commented 6 years ago

cc'ing @hroncok as well.

hroncok commented 6 years ago

Funny thing:

(I have Fedora 27 with all the BuildRequires from our python-setuptools packages installed (the missing ones I got from Fedora 28.)

$ python3 -m venv __venv__
$ . __venv__/bin/activate
(__venv__) $ python -m pip install -r tests/requirements.txt
(__venv__) $ PYTHONPATH=$(pwd) python -m pytest -k easy_install
all good
(__venv__) $ deactivate
$ python3 -m venv __venv__ --system-site-packages
$ . __venv__/bin/activate
(__venv__) $ PYTHONPATH=$(pwd) python -m pytest -k easy_install
boom

This makes me think that something from my system is interfering with the tests. Will try to find out what. (EDIT: It was RPM installed pytest-flakes.)

hroncok commented 6 years ago

using --assert=plain workarounds the issue, PYTHONDONTWRITEBYTECODE=1 also workarounds this issue

pytest is calling os.mkdir(cache_dir) when setuptools.sandbox is imported. Using --assert=plain makes it not do "assert magic" at all, using PYTHONDONTWRITEBYTECODE=1 makes it not store the files and call mkdir. Setuptool's DirectorySandbox is trying to prevent the mkdir call. When it does, it imports from sandbox again and that triggers pytest to try to do the whole thing from start. Hence the infinite recursion.

Still no idea why it blows up only when system site packages are allowed :(

hroncok commented 6 years ago

pytest's AssertionRewritingHook._must_rewrite:

with system site packages: {'pytest_virtualenv', 'pytest_shutil', 'pytest_flake8', 'setup', 'setuptools.tests.fixtures'} (notice setup)
without system site packages: {'pytest_shutil', 'pytest_virtualenv', 'pytest_flake8', 'setuptools.tests.fixtures'}

as setuptools.sandbox startswith setup, this is causing a problem later at https://github.com/pytest-dev/pytest/blob/77bd0aa02ffc7f8734682c2e353135af5d94ab76/_pytest/assertion/rewrite.py#L170

hroncok commented 6 years ago

I have found the difference with system site-packages:

import pprint
import pkg_resources

# from pytest's _mark_plugins_for_rewrite
metadata_files = 'RECORD', 'SOURCES.txt'
package_files = (
    entry.split(',')[0]
    for entrypoint in pkg_resources.iter_entry_points('pytest11')
    for metadata in metadata_files
    for entry in entrypoint.dist._get_metadata(metadata)
)

pprint.pprint(list(package_files))

If you install a pytest plugin (such as pytest-virtualenv or pytest-flake8) from a wheel, the list does not include setup.py and pytest.cfg. If you however install trough sdist, it does. This puts setup to pytest's AssertionRewritingHook._must_rewrite.

A reproducer:

$ python3 -m venv __venv_reproducer__
$ . __venv_reproducer__/bin/activate

(__venv_reproducer__) $ pip install https://pypi.python.org/packages/e5/87/345c1423d3dd7c27247b61c71192c9b94a5f980647389e622ac41ff92a3d/pytest-flake8-0.9.1.tar.gz#md5=f16c46d0fa88e07606b0c444d697d815
...
Successfully installed flake8-3.5.0 mccabe-0.6.1 py-1.5.2 pycodestyle-2.3.1 pyflakes-1.6.0 pytest-3.2.5 pytest-flake8-0.9.1

(__venv_reproducer__) $ pip install -r tests/requirements.txt 
...
Requirement already satisfied: pytest-flake8 in ./__venv_reproducer__/lib/python3.6/site-packages (from -r tests/requirements.txt (line 3))
...
Successfully installed apipkg-1.4 contextlib2-0.5.5 execnet-1.5.0 mock-2.0.0 path.py-10.5 pbr-3.1.1 pytest-fixture-config-1.2.11 pytest-shutil-1.2.11 pytest-virtualenv-1.2.11 six-1.11.0 virtualenv-15.1.0

(__venv_reproducer__) $ PYTHONPATH=$(pwd) python -m pytest -k test_unicode_filename_in_sdist -v --tb=line
...
setuptools/tests/test_easy_install.py::TestEasyInstallTest::test_unicode_filename_in_sdist FAILED
...
<frozen importlib._bootstrap>:968: RecursionError: maximum recursion depth exceeded
...

Our RPM packages are installed using python setup.py install, not trough wheels, that's why we are hitting this problem.

hroncok commented 6 years ago

I believe this is a pytest bug.

_pytest.config._mark_plugins_for_rewrite should not register setup when it sees setup.py
_pytest.assertion.rewrite._should_rewrite should probably match things that startswith mark + '.', not just mark (although I'm not very sure here)

Could you please help here @nicoddemus? This comes from https://github.com/pytest-dev/pytest/commit/6711b1d6abf3adc2356e3b58c0b0f664bbbb77d2

nicoddemus commented 6 years ago

Hi @hroncok, thanks for the investigation!

Indeed your assessment seems correct. Created pytest-dev/pytest#2939 to track it, I will try to get to it soon.

nicoddemus commented 6 years ago

pytest-dev/pytest#2939 has been fixed and will be published in the next pytest release. Should we close this given that there's nothing for setuptools to do about it?

hroncok commented 6 years ago

Fedora package has been "fixed" by using PYTHONDONTWRITEBYTECODE=1. I agree we can close this (I cannot close).

stratakis commented 6 years ago

Kudos to everyone involved in the investigation of this issue!

pypa / setuptools

Updating to 36.5.0 breaks tests #1170