Closed cfm closed 2 years ago
The key part of the log seems to be:
Installing build dependencies ... - \ | / - \ | / - \ done
Getting requirements to build wheel ... - done
Installing backend dependencies ... - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done
Preparing metadata (pyproject.toml) ... - error
ERROR: Command errored out with exit status 1:
command: /home/ci/project/.venv/bin/python3 /home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpa46sr7yq
cwd: /tmp/pip-download-uomjt3pc/python-dateutil_f166952022704c91aa4cd2f016a88be2
Complete output (32 lines):
Traceback (most recent call last):
File "/home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 154, in prepare_metadata_for_build_wheel
backend = _build_backend()
File "/home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
obj = import_module(mod_path)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/tmp/pip-build-env-52am7k_j/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 18, in <module>
from setuptools.dist import Distribution
File "/tmp/pip-build-env-52am7k_j/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 35, in <module>
from ._importlib import metadata
File "/tmp/pip-build-env-52am7k_j/overlay/lib/python3.7/site-packages/setuptools/_importlib.py", line 39, in <module>
disable_importlib_metadata_finder(metadata)
File "/tmp/pip-build-env-52am7k_j/overlay/lib/python3.7/site-packages/setuptools/_importlib.py", line 30, in disable_importlib_metadata_finder
for ob in sys.meta_path
File "/tmp/pip-build-env-52am7k_j/overlay/lib/python3.7/site-packages/setuptools/_importlib.py", line 31, in <listcomp>
if isinstance(ob, importlib_metadata.MetadataPathFinder)
AttributeError: module 'importlib_metadata' has no attribute 'MetadataPathFinder'
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/0e/01/68747933e8d12263d41ce08119620d9a7e5eb72c876a3442257f74490da0/python-dateutil-2.7.5.tar.gz#sha256=88f9287c0174266bb0d8cedd395cfba9c58e87e5ad86b2ce58859bc11be3cf02 (from https://pypi.org/simple/python-dateutil/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*). Command errored out with exit status 1: /home/ci/project/.venv/bin/python3 /home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpa46sr7yq Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement python-dateutil==2.7.5 (from versions: 1.4, 1.4.1, 1.5, 2.1, 2.2, 2.3, 2.4.0, 2.4.1, 2.4.1.post1, 2.4.2, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.7.5, 2.8.0, 2.8.1, 2.8.2)
ERROR: No matching distribution found for python-dateutil==2.7.5
What does this mean? (Using a diff just to highlight the parts that surprise me):
ERROR: Could not find a version that satisfies the requirement python-dateutil==2.7.5 (from versions: 1.4, 1.4.1, 1.5, 2.1, 2.2, 2.3, 2.4.0, 2.4.1, 2.4.1.post1, 2.4.2, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.7.5, 2.8.0, 2.8.1, 2.8.2)
+ ERROR: ------------------------------------------------------------------------>2.7.5 (from versions: -keep scrolling-------------------------------------------------------------------------------------------------------------------------------->2.7.5<--------------------)
Nothing significant, it's just how pip expresses the error. Because 2.7.5 couldn't be installed, obviously none of the other versions on pypi work since we're pinned to 2.7.5.
I did a bit of searching, I think bumping us up to 2.8.0 might fix it (https://dateutil.readthedocs.io/en/latest/changelog.html#id6 - addition of pyproject.toml) but also didn't finish looking into what that would take.
I tried reproducing this locally and couldn't... :thought_balloon: not sure why.
cd ~/src/securedrop-client
git checkout main
cp ~/src/securedrop-client /tmp/securedrop-clientlh0mh054
cd ~/src/securedrop-debian-packaging
git checkout release-securedrop-workstation-config # known to be failing in CI
pip3 download --no-binary :all: --require-hashes --dest /tmp/tmp8zgdeb0_ --requirement /tmp/securedrop-clientlh0mh054/requirements/requirements.txt
# exits successfully within a few minutes
To unblock #355, #356, and #357 (with more to come), I started to update securedrop-client
's requirements for python-dateutil
2.8.0. However, quick testing reveals that neither v2.7.5 and v2.8.0 installs under Python 3.7 on Debian buster, while both install fine under Python 3.9 on Debian bullseye. Sure enough, reprotest-wheels
's prebaked environment dates from last July on Debian buster. So updating that image for Debian bullseye has got to be our first step, which I'll start now.
In preparation for next week's retrospective on this problem in the Workstation release process, @creviera has asked me to look more deeply into why the python-dateutil==2.7.5
dependency chain began failing for us under buster but not under bullseye. I'll close this ticket again next week with my findings.
My theory was that something in pip/setuptools/wheel/etc. was automatically upgraded and prevented the old version of the package from installing. But that doesn't really address why it started working on bullseye. I would also like to get rid of the "builder" image, it doesn't actually add any value here...
I've been able to construct a minimal reproduction of this failure, revealing that:
Specifically, reprotest-wheels
began to fail under Python 3.7 with setuptools_scm==7.0.1
, which added an explicit dependency for
importlib-metadata;python_version < '3.8'
We would have needed to wait for an upstream fix for python/importlib_metadata#392 to restore Python 3.7 compatibility, but we were able to sidestep that requirement by switching reprotest-wheels
to newly-preferred Python 3.9.
@cfm ooh, nice discoveries. Was the dependency not properly pinned?
Good question. It's not obvious to me what to pin. I've just updated the reproduction to pin all of setuptools
, setuptools_scm
, and importlib-metadata
, and the build-time circularity described in https://github.com/pypa/packaging-problems/issues/342#issuecomment-1166304390 remains under Python 3.7.
After today's retrospective, I'm working on a simplified reproduction, demonstrating the interactions of just setuptools_scm
and importlib-metadata
under Python 3.7....
Let's do some Socratic troubleshooting based on this even-narrower reproduction....
Why does python-dateutil==2.7.5
fail to install under Python 3.7? Because it fails to build from source, as of 22 June 2022.
Why does it fail to build from source? Because its backend dependency setuptools_scm
fails to build from source. (pypa/setuptools_scm#722)
Why does setuptools_scm
fail to build from source? Because, under Python < 3.8, its backend dependency importlib-metadata
fails to build from source. (python/importlib_metadata#392)
Why does importlib-metadata
fail to build from source? Because it depends on setuptools_scm
....
When was this change introduced in setuptools_scm
? Version 7.0.1, 21 June 2022.
Can't we pin to setuptools_scm<7.0.1
Sure. But this will still fail:
pip install --no-binary :all: "setuptools_scm<7.0.1" python-dateutil==2.7.5
Wait, why is importlib-metadata
trying to install setuptools_scm==7.0.5
when it already has setuptools_scm<7.0.1
? Because pip
is installing importlib-metadata
's build-system.requires
dependencies under "build isolation", ignoring the setuptools_scm
we've already installed.
But aren't we specifying setuptools_scm<7.0.1
in the same requirements specification as python-dateutil==7.0.1
? Yup.
What if we do pip install -c constraints.txt [...]
with setuptools_scm<7.0.1
in constraints.txt
? Nope, because "generated pip
commands do not respect constraints files in outer install command".
So what about PIP_CONSTRAINT=constraints.txt pip install [...]
? YES!
What does this mean for us? If we want fully deterministic dependencies—including, for reproducible wheels, fully deterministic build dependencies—then we must pin these either:
pip-compile --allow-unsafe
'd requirements.txt
, enforced in CI via PIP_CONSTRAINT=requirements.txt pip install -r requirements.txt
; orconstraints.txt
or build-requirements.txt
, enforced in CI in the same way.We started doing this in securedrop
in freedomofpress/securedrop#4686. I'll open a follow-up ticket to consider options (1) and (2) for securedrop-client
, since—even if #358 let us sidestep this presenting problem under under Python 3.7—we've been bitten by this constraint-enforcement problem in the past and surely will be bitten by it again.
Specifically, installing
python-dateutil
forsecuredrop-client
in:https://github.com/freedomofpress/securedrop-debian-packaging/blob/b13e30d06d01a43b4909434f1cb38017b0dec951/scripts/build-sync-wheels#L75-L90
Evident in CI (e.g.) and reproducible by invoking manually:
Originally posted by @cfm in https://github.com/freedomofpress/securedrop-debian-packaging/issues/349#issuecomment-1164649957