Open nitzmahone opened 1 year ago
In a case where 5.4.1 arrives as a dependency from requirements.txt the workaround for us looks like this. Unfortunately when we build just the wheel with PIP_CONSTRAINT, that wheel is not used when installing from requirements.txt.
echo 'Cython < 3.0' > /tmp/constraint.txt
PIP_CONSTRAINT=/tmp/constraint.txt pip install 'PyYAML==5.4.1'
pip install -r requirements.txt
Thanks for posting the workaround. It was very helpful.
@leiflinse-trivector Interesting- we (Ansible) have successfully deployed this workaround extensively throughout our CI infra on dozens of different OSs from Python 2.7-3.12 and numerous pip versions, just as described above. One caveat I purposely omitted from the description above (since I assumed people were only building PyYAML for platforms/Pythons that didn't already have a wheel): if the index server pip consults has an "applicable" PyYAML wheel, it seems to prefer the wheel on the index server to the local one. This seems to happen more if the PyYAML requirement is unconstrained by the dependent package- the presence of any exact or upper-bound constraint on PyYAML seems to make the locally-cached wheel eligible for use (eg, if the dependent package asks for PyYAML==5.4.1
or PyYAML<6
)- if the dependent package doesn't specify an upper bound and is incompatible with PyYAML 6.x, it's already arguably broken anyway, but in that case, you can also use the PIP_CONSTRAINT
trick to augment the broken package/requirements install with your own constraint for eg, PyYAML<6
that should cause it to use the locally-cached wheel.
Is there a similar workaround for pipenv
? Honestly this is what I'm doing but not sure I like it...
pip3 install "cython<3.0" wheel && pip3 install --no-build-isolation "pyyaml==5.4.1" && \
pipenv sync --system
@reid-harrison No idea here- not a pipenv user. That said, all you really need is a locally-built 5.4.1 wheel that's been constrained as necessary, so why not just do the pip wheel
as describe above to create the wheel, then just install the wheel that pip created in CWD with pipenv
as normal?
On Windows the commands should look like this
# create a constraint.txt file that limits the Cython version to one that should work
# for example in C:\, with this contents:
Cython < 3.0
# seed pip's local wheel cache with a PyYAML wheel
set PIP_CONSTRAINT=C:\constraint.txt & pip wheel PyYAML==5.4.1
# install PyYAML itself, or any other package(s) that ask for the PyYAML version you just built
py -m pip install 'PyYAML==5.4.1'
@leiflinse-trivector You probably should describe your environment better to make your comment more useful. If people can reproduce your case, they might come up with a better workaround.
I didn't test it extensively, but I needed to run docker-compose<2.x
in a docker
container. Here's the script:
It can probably be improved and doesn't handle all the cases, but the workaround works in this case.
This worked for me in a python 3.12 poetry install (my other dependencies wanted <6
:-(, so this ended up as 5.3.1):
pyyaml = { version = "!=6.0.0,!=5.4.0,!=5.4.1" }
Thank you. Since there seems to be a bit of confusion around pipenv, I want to confirm that following the first solution verbatim and then running pipenv install
afterward fixes the issue.
Just a warning to maintainers: PyYAML supporting Python 3.13 is supposedly going to require switching to Cython 3.
@nanonyme Yeah, I've been doing all my local 3.13 testing against Cython 3 with the old_build_ext
stuff and it's been working fine. We'll probably end up implementing at least conditional support for that in the update that supports 3.13, since the "shred the extension build to ditch all the distutils-isms" thing is still in flux.
@nanonyme [...] supporting Python 3.13 is supposedly going to require switching to Cython 3.
My understanding is this is due to python/cpython#104775 (CPython 3.13+) and cython/cython#5128 (Cython 0.29.36+).
@nitzmahone [...] the "shred the extension build to ditch all the distutils-isms" thing is still in flux.
Care to elucidate what is holding back the jettison of distutils
APIs in #797 (but feel free to comment over there where it is probably more pertinent to the discussion)?
In the instructions below, replace
pip
as needed with apip
invocation matching your target Python environment (eg,pip3
,/usr/local/bin/python3.11 -m pip
).Pre-seed wheel cache (recommended)
This solution pre-seeds pip's wheel cache with a locally-built PyYAML wheel, accessible to any subsequent installation using the same pip cache.
Inline constraint (simpler, more likely to break)
This solution globally constrains the Cython version for all packages being installed by this pip invocation (including nested/child package installs), which could break other packages installed at the same time.
Background
With the release of Cython 3, all older versions of PyYAML can no longer be installed from unmodified source or sdist (ie, where a wheel is unavailable for the platform and/or Python), since the Cython version was not capped to a working version for all older PyYAML releases. For various reasons, it is untenable to release new sdists/wheels for these old PyYAML versions with the new required
Cython<3.0
build dependency constraint.pip
has mostly-undocumented support for "inherited" constraints at install-time by setting thePIP_CONSTRAINT
environment variable (which is inherited by child build processes, unlike the CLI--constraint
arg).Sample error output from
pip install
If you're seeing an error similar to this when installing a version of PyYAML older than 6.0.1, the preceding solutions may help.