Open jaraco opened 1 year ago
In 51408ac9f34f8ca97c33f7f234160c502acb11c4, I disabled the cygwin job, but then the builds started failing on Python 3.12 where they didn't previously. I don't understand why. Does the codebase no longer work with Python 3.12 in non-main branches?
Hi @jaraco, I noticed that the skeleton
project does not have any skeleton
folder or skeleton.py
that can be imported when the package is installed. Nevertheless, the docs still specify .. automodule:: skeleton
.
Could it be the case a new version of sphinx
(version 7.0.0 was released on Apr 29) behaves differently when it cannot import a package? Or maybe sphinx 7.0.0
has some conditional logic for automodule in Python 3.12?
If I run:
> docker run --rm -it python:3.12.0a7-bullseye /bin/bash
git clone https://github.com/jaraco/skeleton /tmp/skeleton
cd /tmp/skeleton
python3.12 -m venv .venv
.venv/bin/python -m pip install -e '.[docs,testing]'
cd docs
../.venv/bin/python -m sphinx -W --keep-going . _build/html
# ...
# WARNING: autodoc: failed to import module 'skeleton'; the following exception was raised:
# No module named 'skeleton'
# ...
# build finished with problems, 1 warning
I can see the error happening.
However if I manually add a skeleton
directory before installing, Sphinx does not show an error:
> docker run --rm -it python:3.12.0a7-bullseye /bin/bash
git clone https://github.com/jaraco/skeleton /tmp/skeleton
cd /tmp/skeleton
mkdir skeleton
python3.12 -m venv .venv
.venv/bin/python -m pip install -e '.[docs,testing]'
cd docs
../.venv/bin/python -m sphinx -W --keep-going . _build/html
# ...
# build succeeded.
# The HTML pages are in _build/html.
I se I must have confused myself. I didn't mean to be looking at the pipelines on skeleton but on Setuptools. Here I'm mainly concerned about making sure the Setuptools pipelines are passing.
Error is here:
setuptools.__version__
(Ref: https://github.com/pypa/setuptools/actions/runs/4943028741/jobs/8837121715)
It is been a while that we are experiencing flaky tests erroring in things related to importlib.metadata
.
I think they first started appearing with the change in importlib-metadata v5.2.0 (vendored in setuptools v67.4.0).
@jaraco, after some investigation, I suspect that the problem is the following:
pytest-xdist
(for the sake of speed)setuptools.build_meta
is called by different tests to build setuptools itself.setuptools.build_meta
calls egg_info
a few times (for each one of the PEP 517 hooks, including get_requires*
).
.egg-info/PKG-INFO
file is rewritten everytime egg_info
is calledegg_info.egg_base
parameter is set to the project root (therefore there is no isolation for egg_info
, if multiple builds happen concurrently, all of them will try to write the same files at the same time). This problem is related to https://github.com/pypa/setuptools/issues/3119.PKG-INFO
file is rewritten, the tests will try to use importlib_metadata
(e.g. to read version, entry-points, to find plugins for pytest/flake8, etc..), which in turn will try to read the PKG-INFO
(because setuptools.egg-info
is in sys.path
, due to the flat layout in the repository).egg_info
and importlib_metadata
which results in the PKG-INFO
file being empty.Right now we don't have a way to avoid running egg_info
, and moreover when I tried to change setuptools.build_meta
to set egg_base
to a temporary directory, I had bootstrapping issues.
The bootstrap issues occur because bootstrap.egg-info/entry_points.txt
only have the minimum necessary to run egg_info
, and the whole bootstrapping process accidentally rely on the fact that egg_info
will generate another setuptools.egg-info/entry_points.txt
in the project root file before other PEP 517 hooks are called (so when build_sdist
and build_wheel
are called, importlib_metadata.entry_points
can find the final entrypoints specified by setuptools in its setup.cfg
).
In https://github.com/pypa/setuptools/pull/3904, when I was investigating this problem I tried to improve the way the PKG-INFO
files are generated to be more "atomic" (https://github.com/pypa/setuptools/pull/3904/commits/ae8ec5c3f4cfc7f14783b4dd58c436a7ba168138). This should be an improvement but it is not the whole solution.
I also run into the setuptools.__version__
problem and added a workaround in #3915.
I also run into the
setuptools.__version__
problem and added a workaround in #3915.
Aha. I was sure I'd synced to the latest main before working on this issue, but it seems I had not (51408ac9 is based on an old parent).
bootstrapping process accidentally rely on
I wouldn't say it's accidental. I'd intended for the bootstrapping to provide the minimum customization to allow setuptools to run normally (and minimize duplication).
In d2ec0473f8d4c25cc6f696e70ba110e1061e4dfe, I replaced pytest-flake8 with pytest-ruff (jaraco/skeleton#79). Unfortunately, adding ruff implies adding a dependency on Rust :( and the test_cygwin job is failing in CI because it doesn't have rust.