Request for a v5.4.1.post1 release to set `Cython<3`

sirosen commented 11 months ago

I would like to propose and hear pyyaml maintainers' feedback on the suggestion that a new release be cut off of the 5.4.1 release, the only change being an update to build-system.requires in pyproject.toml to list Cython<3.

This issue is intended to separate discussion of this potential workaround from all of the other discussion of the Cython 3.0 release impact, pyyaml 6.x, etc.

Why do this?

It's important to answer why this particular workaround is valuable, rather than a few of the other options.

Upgrading to pyyaml 6.0.1+
Installing Cython 2.x into the build environment explicitly

First, to the matter of upgrading. This is likely the best approach for any code over which pyyaml users have direct control. Unfortunately, some of us are working with second or third-order dependencies or other situations in which we have limited control. I've followed this a bit in my own case and found myself at an EOL project which pins pyyaml<6. Just as I want my dependencies to upgrade, I'd like for my dependencies to move off of EOL libraries, but this is the unfortunate reality of some existing toolchains.

Second, controlling the build environment. This is possible, but it gets harder the further you are, as a user, from the direct pip invocation used to do the install. In particular, tox installation of deps and poetry installation of dependencies have been noted among end-users of pyyaml-based projects as situations in which the --no-build-isolation workaround isn't highly feasible.

How can this get done?

Basically, there needs to be a new branch for 5.4.1.post1, a commit on that branch with the Cython<3 fix, and then whatever the "normal" release process looks like for pyyaml (twine upload, etc).

It looks like the build infrastructure has advanced a bit in the intervening 2 years, but not so much that a release should be impossible.

There isn't very much room for community involvement on this. I could create a branch off of 5.4.1 with the necessary build-system update, but there isn't a valid PR target for such a change.

Will this imply that 5.x is still maintained? Why 5.4.1.post1 rather than 5.4.2?

No. This risks getting into somewhat ideological territory about how package maintainers behave, but I don't think any reasonable direct user of pyyaml would read a 2-years-after-the-fact fix to make installs possible as a signal that 5.x is now safe to use.

This is also why I propose doing this as a post release. There is very little difference between 5.4.1.post1 and 5.4.2 other than the cosmetics. PEP 440 specifies that 5.4.1.post1 sorts after 5.4.1, which is all we need for this fix to be effective.

aeurielesn commented 11 months ago

@nitzmahone answered here.

sirosen commented 11 months ago

I hadn't seen that PR; thanks for the link! That comment explains the situation much better than some of the other threads.

To extract some specific detail from that,

The packaging metadata is the easy part- the hard part is rebuilding and re-releasing all the same wheels without risking breakage (esp since a number of the worker images have been discontinued). If we release a new 5.x sdist, we have to re-release all the wheels or we've broken everyone else that was still working fine with those old versions.

This speaks to the issue as seen from the perspective of not wanting to break existing (working) installs of 5.4.1 wheels by releasing new/different packages from newer infrastructure.

I think that rules out the possibility of using 5.4.2 or 5.4.1.post1. I don't like that we're in this situation, but now that we're here, it doesn't seem productive to continue considering those options.

What about the possibility of publishing a dev or alpha release such that it can be explicitly selected?

If 5.4.2.dev1 were released with all new wheels from new infrastructure -- and only for current pythons -- it would be possible for downstream users to build their environments by explicitly specifying

pip install 'sad-third-party-tool-which-depends-on-pyyaml-five' 'pyyaml==5.4.2.dev1'

That doesn't risk breaking anyone except those who are relying on pip install --pre. It's a much narrower impact, and only for users who are already explicitly opting in to some relatively unstable behavior.

If the backported pre-release idea isn't viable, or simply too unsightly to consider, I'd ask a maintainer to simply close this issue.

SoftwareApe commented 11 months ago

Alternatively I'd suggest fixing the setup.py to work with Cython3 https://github.com/yaml/pyyaml/pull/729.

EDIT: Closed #729 in favor of #731. Please apply it to 5.4.1 release branch as well, since we need a fix there.

Uzume commented 11 months ago

Instead of explicitly asking for a known workaround (i.e., Cython < 3 dependency as in #726 and #702), this issue should be asking for a resolution of #601 for a 5.x release (e.g., 5.4.1.post1).

Perhaps a better solution for this issue would be to close it as a duplicate of #724 (which reports it as a more generic issue on 5.x).

I look forward to a #731 (and formerly #602) type of fix for a 5.x release.

idanmiara commented 11 months ago

I'm not entirely sure, but I think that PyYAML~=5.4.1 would pick up 5.4.2 but not 5.4.1.x. So I think we should keep the versioning simple with 5.4.2.

sirosen commented 11 months ago

Post releases and dev releases are both well defined by pep 440 and are supported by the mainstream installers and solvers.

If you have a third order dependency which does exact pinning (==) then no 5.x release can help you, but any looser specifier in the <6 category will suffice. e.g. If you have a dependency which asks for ~=5.4.1 and your package explicitly depends on ==5.4.2a1, then there is a (exactly one) solve for your pyyaml version.

At this point, there isn't very much point in discussing the finer points of the version semantics. The pyyaml maintainers first need to demonstrate some interest in this approach.

nitzmahone commented 11 months ago

I'm less opposed to doing an sdist-only pre-release of 5.4.1 with the packaging metadata capped at Cython<3.0 (as that would mostly sidestep the "we can't rebuild all the 5.x wheels anymore" problem), but I can still see some problems with that approach, and it still seems worse than just blocking Cython 3 via PIP_CONSTRAINTS in pretty much every case:

anyone doing pip install $thing_using_old_pyyaml --pre will suddenly have no wheels
dependent projects that explicitly link to the new pre-release for non-wheel users would lose wheels for everyone else
how would someone discover that they need to do this?
dependent projects pinned to a specific PyYAML version would either require a re-release to relax their pins or the use of PIP_CONSTRAINTS or some other mechanism to get the updated one. If the dependent project has to do a release anyway, why wouldn't they just make the sub-one-line code change to support PyYAML >= 6.0? On the flip side, if you're doing PIP_CONTSTRAINTS, why not just specify Cython<3.0 there and call it a day?

Long answer short: we can do a 5.4.1.1rc1 or something with the metadata fix (that would never actually be "released") , as the new problems it creates for all the existing 5.x users that are working fine today are minimal enough that I can probably live with them, but I still don't think it's going to make anyone happy.

nitzmahone commented 11 months ago

PS: if you don't follow Python packaging carefully, the entire Python ecosystem will likely start having this problem x1000 in the next few months, when all the projects that have uncapped build-time setuptools deps start breaking as setuptools begins aggressively removing deprecated functionality.

sirosen commented 11 months ago

I feel like I've somehow not understood something regarding the wheel builds.

Let's suppose we agree that the new version will be 5.4.1.1rc1 -- just to set the exact version string aside but agree that it's a prerelease. Why does that mean that no wheels can be built at all? Shouldn't that just mean doing wheel builds for all of the platforms which were supported by 5.4.1 and 6.x? By my accounting, that's 3.6, 3.7, 3.8, and 3.9 on manylinux (several flavors), win32, win64, and macosx_10_9.

I understand not wanting to try to move 5.x to support the platforms supported by 6.x or other aggressive efforts -- the goal I have here is to be as narrow as possible and demand the least from maintainers to get wedged users unstuck. But can we not build a very strict subset of the wheels where the builds are possible? I also don't think the badness of dropping wheels for those users who are already using --pre and are otherwise in an "okay" situation today on 5.x is severe. They can update to set pyyaml==5.4.1 in their requirements in the worst case.

Regarding two specific points above, I think there are easy answers:

dependent projects pinned to a specific PyYAML version would either require a re-release to relax their pins ...

Yep, there's no improvement for these cases, but I consider them completely out of scope. I'm mostly concerned with dependency chains which include completely dead projects which specify pyyaml<6. No release or update to the source repo or other change will ever happen for these, but they haven't done the most harmful thing possible by pinning an exact version.

how would someone discover that they need to do this?

Mostly by bumping into these threads on github and it becoming the canonical answer. I don't think it's a problem that it's obscure or arcane. The whole use of a prerelease for this already indicates that we're playing packaging games to solve the problem -- it's not a beautiful solution, just a workaround.

Plus, the pyyaml maintainers get to close all of these related discussions aggressively with some standard-ish text of "You can use 5.4.1.1rc1 and see if that works for you. If it doesn't, that's as far as we're willing to go to accommodate these old toolchains, so you'll need to find a way to constrain the cython version in your build environment."

idanmiara commented 11 months ago

Long answer short: we can do a 5.4.1.1rc1 or something with the metadata fix (that would never actually be "released") , as the new problems it creates for all the existing 5.x users that are working fine today are minimal enough that I can probably live with them, but I still don't think it's going to make anyone happy.

What about making 5.4.2 sdist but for Python>=3.10 only ? https://github.com/yaml/pyyaml/issues/724#issuecomment-1647943670 This way nobody would need to change a thing in their pipelines (maybe only whoever uses Python<=3.9 for platforms that wheels were not released for? I'm guessing that is a far less common case).

sirosen commented 11 months ago

sdist-only for 3.10+ is an interesting idea. If there's some concern about the <=3.9 users who don't have available wheels, one option is to provide two releases in series, one as a prerelease:

set cython<3.0 on the 5.x branch
5.4.2rc1: sdist-only, no python_requires
5.4.2: sdist-only, python_requires>=3.10

As long as there's at least one version of pyyaml<6 which is installable without manually constraining the build environment, a user can cut through a wide swath of issues by making a specific pyyaml version one of their direct dependencies.

nitzmahone commented 11 months ago

Can you try the workaround in #736? Python-version limited releases don't solve a lot of the problems we're likely to face here...

idanmiara commented 11 months ago

Can you try the workaround in #736? Python-version limited releases don't solve a lot of the problems we're likely to face here...

I think that the workarounds mostly do work, but require all the effected projects to apply they. Could you please elaborate which problems the proposed solution by @sirosen won't solve?

nitzmahone commented 11 months ago

Closing- see proposed workarounds. We won't be accepting any backports or doing any releases for older PyYAML branches.

yaml / pyyaml