Open jaraco opened 1 year ago
jaraco
commented
1 year ago In 51408ac9f34f8ca97c33f7f234160c502acb11c4, I disabled the cygwin job, but then the builds started failing on Python 3.12 where they didn't previously. I don't understand why. Does the codebase no longer work with Python 3.12 in non-main branches?
abravalheri
commented
1 year ago Hi @jaraco, I noticed that the skeleton project does not have any skeleton folder or skeleton.py that can be imported when the package is installed. Nevertheless, the docs still specify .. automodule:: skeleton.
Could it be the case a new version of sphinx (version 7.0.0 was released on Apr 29) behaves differently when it cannot import a package? Or maybe sphinx 7.0.0 has some conditional logic for automodule in Python 3.12?
If I run:
> docker run --rm -it python:3.12.0a7-bullseye /bin/bash
git clone https://github.com/jaraco/skeleton /tmp/skeleton
cd /tmp/skeleton
python3.12 -m venv .venv
.venv/bin/python -m pip install -e '.[docs,testing]'
cd docs
../.venv/bin/python -m sphinx -W --keep-going . _build/html
# ...
# WARNING: autodoc: failed to import module 'skeleton'; the following exception was raised:
# No module named 'skeleton'
# ...
# build finished with problems, 1 warningI can see the error happening.
However if I manually add a skeleton directory before installing, Sphinx does not show an error:
> docker run --rm -it python:3.12.0a7-bullseye /bin/bash
git clone https://github.com/jaraco/skeleton /tmp/skeleton
cd /tmp/skeleton
mkdir skeleton
python3.12 -m venv .venv
.venv/bin/python -m pip install -e '.[docs,testing]'
cd docs
../.venv/bin/python -m sphinx -W --keep-going . _build/html
# ...
# build succeeded.
# The HTML pages are in _build/html.I se I must have confused myself. I didn't mean to be looking at the pipelines on skeleton but on Setuptools. Here I'm mainly concerned about making sure the Setuptools pipelines are passing.
Error is here:
abravalheri
commented
1 year ago setuptools.__version__(Ref: https://github.com/pypa/setuptools/actions/runs/4943028741/jobs/8837121715)
It is been a while that we are experiencing flaky tests erroring in things related to importlib.metadata.
I think they first started appearing with the change in importlib-metadata v5.2.0 (vendored in setuptools v67.4.0).
@jaraco, after some investigation, I suspect that the problem is the following:
pytest-xdist (for the sake of speed)setuptools.build_meta is called by different tests to build setuptools itself.setuptools.build_meta calls egg_info a few times (for each one of the PEP 517 hooks, including get_requires*).
.egg-info/PKG-INFO file is rewritten everytime egg_info is calledegg_info.egg_base parameter is set to the project root (therefore there is no isolation for egg_info, if multiple builds happen concurrently, all of them will try to write the same files at the same time). This problem is related to https://github.com/pypa/setuptools/issues/3119.PKG-INFO file is rewritten, the tests will try to use importlib_metadata (e.g. to read version, entry-points, to find plugins for pytest/flake8, etc..), which in turn will try to read the PKG-INFO (because setuptools.egg-info is in sys.path, due to the flat layout in the repository).egg_info and importlib_metadata which results in the PKG-INFO file being empty.Right now we don't have a way to avoid running egg_info, and moreover when I tried to change setuptools.build_meta to set egg_base to a temporary directory, I had bootstrapping issues.
The bootstrap issues occur because bootstrap.egg-info/entry_points.txt only have the minimum necessary to run egg_info, and the whole bootstrapping process accidentally rely on the fact that egg_info will generate another setuptools.egg-info/entry_points.txt in the project root file before other PEP 517 hooks are called (so when build_sdist and build_wheel are called, importlib_metadata.entry_points can find the final entrypoints specified by setuptools in its setup.cfg).
In https://github.com/pypa/setuptools/pull/3904, when I was investigating this problem I tried to improve the way the PKG-INFO files are generated to be more "atomic" (https://github.com/pypa/setuptools/pull/3904/commits/ae8ec5c3f4cfc7f14783b4dd58c436a7ba168138). This should be an improvement but it is not the whole solution.
I also run into the setuptools.__version__ problem and added a workaround in #3915.
jaraco
commented
1 year ago I also run into the
setuptools.__version__problem and added a workaround in #3915.
Aha. I was sure I'd synced to the latest main before working on this issue, but it seems I had not (51408ac9 is based on an old parent).
bootstrapping process accidentally rely on
I wouldn't say it's accidental. I'd intended for the bootstrapping to provide the minimum customization to allow setuptools to run normally (and minimize duplication).
In d2ec0473f8d4c25cc6f696e70ba110e1061e4dfe, I replaced pytest-flake8 with pytest-ruff (jaraco/skeleton#79). Unfortunately, adding ruff implies adding a dependency on Rust :( and the test_cygwin job is failing in CI because it doesn't have rust.