search

pypa / manylinux

Python wheels that work on any linux (almost)
MIT License
1.44k stars 218 forks source link

Tracking issue for manylinux_2_28 image #1282

Closed mattip closed 2 years ago

mattip commented 2 years ago

Due to issue #1012, the manylinux_2_24 image proposed as the first PEP600 compliant image did not take off. I think we should offer an image based on manylinux_2_28.

The big question is what base image to use. We can choose between C8S (centos:stream8), AlmaLinux, RockyLinux, or UBI. Personally I would prefer an image based entirely on open source software. There is a thread about this on discuss.python.org

base image EOL ¹ gcc-toolset-11 aarch64 ppc64le s390x bug fix availability
CentOS Stream 8 2024-05-31 ² yes yes yes planned ³ 1st
UBI 8 2029-05-31 NO yes yes yes 2nd
AlmaLinux 8 2029-05-31 yes yes yes soon ⁴ 3rd
RockyLinux 8 2029-05-31 yes yes planned ⁵ planned ⁵ 3rd

¹ including Maintenance Support, https://access.redhat.com/support/policy/updates/errata/ ² https://wiki.centos.org/About/Product ³ https://github.com/pypa/manylinux/issues/1282#issuecomment-1038304351 ⁴ https://almalinux.discourse.group/t/plans-for-s390x/875 ⁵ after RockyLinux 9 is out, https://forums.rockylinux.org/t/rocky-linux-8-4-available-now/3015/48

The checklist is short, since wheel and pip, which both use packaging, and warehouse do not require additional PRs.

Publisher-side Support:

Documentation:

Additional projects to check for support for pernnial wheels (left over from gh-542, perhaps these are already done?

tiran commented 2 years ago

We can choose between AlmaLinux, RockyLinux, or UBI.

Is there any particular reason why C8S is missing from the list? centos:stream8 is an option, too. After all AlmaLinux and RockyLinux are getting RPM updates from C8S.

Personally I would prefer an image based entirely on open source software.

All images are entirely based on OSS and freely redistributable. (If you want to be really strict, then you have to exclude all three distributions, because they may contain proprietary firmware blobs.)

mattip commented 2 years ago

Is there any particular reason why C8S

No, I added C8S to the issue description.

All images are entirely based on OSS and freely redistributable

Cool, thanks for the clarification, sorry for any unintentional dispersion that I cast on any project.

mattip commented 2 years ago

@tiran, @kpfleming, @mayeut, @h-vetinari, do you know canonical links to docker images for glibc2_28 linux distributions? Any thoughts as to which distro we should use? I have really no idea how they all relate to each other, and would like to avoid any kind of minefield in our mutual goal to get to a stable base image.

kpfleming commented 2 years ago

We don't need to be concerned about firmware blobs, since only container images are under discussion here :-)

Also, I know it's being pedantic, but the 'container' world consists of more than just Docker in modern times. You may want to consider just referring to the images as "OCI images".

Most of the relation discussion has happened in the Discourse thread. All four image flavors you've mentioned originate from the same source packages (CentOS 8 Stream), and each packager builds their own binary packages.

The UBI8 image is available here. I don't have links for the other images.

h-vetinari commented 2 years ago

do you know canonical links to docker images for glibc2_28 linux distributions?

Assuming we're not talking about canonical-the-company (😉), there's https://hub.docker.com/_/almalinux and https://hub.docker.com/_/rockylinux. @mayeut seems to have experimented with almalinux already.

I don't have a strong preference regarding which distro (I'm guessing one deciding factor might be availability of architecture support like PPC, or perceived viability of the community; here the RH-sponsored flavours obviously have a big advantage...), though I don't understand one aspect about C8S that might be pertinent. Previously, updates flowed from RHEL to CentOS. With stream, this is supposed to be inverted, at least starting with RHEL9.

What I don't know if C8S now lives up- or downstream of RHEL, which - as @tiran and @kpfleming explained - probably matters quite little because the alternatives seems to be using the same sources (though given the history of COS8, I wouldn't be surprised if it's still downstream and just picked up the new "stream" name?).

Where this becomes relevant is if C8S now still is downstream of RHEL (as before), but won't be anymore as of RHEL9 - then having the same name ("centos stream") will imply we should just bump the version 8->9 at some point in the future, when in fact we'd be getting a very different thing. Is what I'm saying understandable, @tiran @kpfleming? Or completely off in some way?

kpfleming commented 2 years ago

If you've got a bit of time, one of our RH colleagues presented a session at FOSDEM 2022 last weekend which shows how the branches and releases relate to each other.

CentOS Stream 8 and CentOS Stream 9 will always be separate, the 'stream' name does not mean that C8S users will just magically get converted to C9S when it is released. It's not a 'rolling' distribution, if that's what you are concerned about.

h-vetinari commented 2 years ago

It's not a 'rolling' distribution, if that's what you are concerned about.

What I'm concerned about is C8S not being a rolling distribution (and thus a viable candidate here), but sharing the name with C9S, which is a rolling distribution (or at least, that's how I understood it so far; I don't necessarily mean "bleeding edge", but by being ahead of RHEL and possibly diverging, it flips the nature of what used to be CentOS on its head).

I would want to avoid a situation where - if we were to choose C8S now - this IMO misleading naming is then used to argue that a future manylinux image should be based on C9S.

kpfleming commented 2 years ago

Understood. I think you can avoid those issues entirely then by just choosing UBI instead of CentOS Stream :)

bookwar commented 2 years ago

What I'm concerned about is C8S not being a rolling distribution (and thus a viable candidate here), but sharing the name with C9S, which is a rolling distribution (or at least, that's how I understood it so far; I don't necessarily mean "bleeding edge", but by being ahead of RHEL and possibly diverging, it flips the nature of what used to be CentOS on its head).

I think there is still some misunderstanding.

CentOS Stream 8 and CentOS Stream 9 share name because they also share the concept. They are two branches of the same thing. So CentOS Stream 9 doesn't "roll" any more than CentOS Stream 8.

CentOS Stream 8 is the representation of RHEL 8 Nightly, CentOS Stream 9 is the representation of RHEL 9 Nightly.

The thing which is different between 8 and 9 currently is that you don't see the merge of the sources happening in 8, you only see the result of it. While in 9 you also see how exactly RHEL developer merges the code for RHEL=CentOS Stream, and how the change passes the RHEL verification stages.

The outcome is still the same: package lands in CentOS Stream 9 after it lands in RHEL 9 Nightly.

h-vetinari commented 2 years ago

OK, thanks for the explanation @bookwar. Can you comment if RHEL9 and C9S will be ABI-locked, or will C9S be allowed to break away and ahead?

h-vetinari commented 2 years ago

(I appreciate the insights; I'd be curious to know what Alma/Rocky will do once C9S rolls around, because the switch to stream created enough FUD that not one, but two projects got launched to fill the perceived void - if RHEL9 and C9S remain fully ABI-compatible, then I don't see much of a functional difference to the old way)

duritong commented 2 years ago

They will stay as ABI compatbile as RHEL minor versions did in the past, since CentOS Stream Y just represents RHEL Y.(N+1) where N is the last minor release of RHEL that got GA. Now when it comes to ABI compatbibility I would refer to RHEL's promise of ABI compatibility within minor releases, since this is essential what CentOS Stream Y will be.

CentOS previously picked up N after it became GA in RHEL and thus could have been lacking (sometimes for months!), now it is the other way around.

bookwar commented 2 years ago

OK, thanks for the explanation @bookwar. Can you comment if RHEL9 and C9S will be ABI-locked, or will C9S be allowed to break away and ahead?

CentOS Stream 9 becomes locked on RHEL 9 ABI compatibility right after RHEL 9 GA.

For pre-release development which is happening now we have the freedom of still changing and adjusting things. Though as RHEL 9 Beta has already been released and we are very close to GA dates there is not going to be any major changes. But after release rules become strict.

h-vetinari commented 2 years ago

OK, thanks for all that - happy to retract my concerns about centos 8 stream in the context of this discussion.

CentOS Stream Y just represents RHEL Y.(N+1) where N is the last minor release of RHEL that got GA

That's the key part IMO. Already previously, the minor version was just that - not very important. Important was (and remains) one cohesive "epoch" represented by the major number + all the ABI stability etc. To be honest, this is a TIL-moment for me, and now I understand way less what the value proposition of alma & rocky is supposed to be.

tiran commented 2 years ago

The official C8S OCI images is hosted on Quay as quay.io/centos/centos:stream8. The image provides developer toolsets like gcc-toolset-11-gcc-11.2.1-1.2.el8_5 out of the box.

carlwgeorge commented 2 years ago

To be honest, this is a TIL-moment for me, and now I understand way less what the value proposition of alma & rocky is supposed to be.

I'll point out that CentOS Stream is the only distribution in the Enterprise Linux family (RHEL and RHEL-like distros) that allows direct contributions to the operating system. Reporting a bug in a RHEL clone like Alma or Rocky will most likely be closed as "reproducible on RHEL, not a bug for us". Their stated goal is to be bug-for-bug compatible with RHEL. Reporting a bug to RHEL will either result in waiting for RHEL maintainers to fix it (in CentOS Stream first) or being guided to contribute the fix to CentOS Stream, which flows into the next RHEL minor release. If the PyPA values being able to contribute to the platform to help shape it to their needs, CentOS Stream is the logical choice.

mayeut commented 2 years ago

Let's add a small comparative table of all the proposed base images in the issue description. I'll try to sum-up and update with all the remarks as they arise.

mayeut commented 2 years ago

I added the comparison table.

EDITED 2022-02-13 19:42 UTC

My 2 cents for how I interpret this table for now:

mattip commented 2 years ago

I would be happy to test an image on NumPy's CI when available

mayeut commented 2 years ago

@mattip, PoC image for x86_64 based on AlmaLinux is available docker pull mayeut/manylinux:manylinux_2_28_x86_64. I will try to get aarch64 & ppc64le PoC images available somehow as well.

mayeut commented 2 years ago

PoC images are now available for testing for x86_64, aarch64 & ppc64le (this last one uses a BETA image from AlmaLinux). I guess whatever the base image we end up choosing, given they all should be compatible for this use-case, it's worth starting getting a bit of feedback (mostly, I'd be interested in : "does auditwheel repair works for your project").

Images repo:

mayeut commented 2 years ago

@messense, I guess it would make sense to add https://github.com/PyO3/maturin the list of tools in the description ?

carlwgeorge commented 2 years ago

CentOS Stream 8: It lacks s390x as do other distros except UBI for now. There are no plans to add it (?) whereas other distros have some.

We plan to add s390x this year as we work towards migrating C8S from the legacy koji instance to the new koji instance, which is part of our plan to align the C8S and C9S workflows. I'm not sure if this has been discussed publicly anywhere yet, but if you're familiar with koji you can see pieces of this being put into place in the new instance, such as tags and build targets.

CentOS Stream 8: It won't receive updates after 2024-05-31 so there will need to be a switch at this point.

I understand this is not appealing, but for whatever it's worth, at that point CentOS Stream 8 and RHEL8 will be basically equivalent as RHEL 8 moves into maintenance support phase. Assuming that UBI8 has all the necessary packages available by then, it should be a seamless transition.

UBI 8: gcc-toolset-11 not available only provides a subset of the packages the other ones provide

Per the UBI FAQ #35, gcc-toolset-11 and anything else required can be requested by opening a bugzilla. I can't make any promises, but if it would make UBI the preferred choice for this, I suspect it would be viewed quite favorably. cc: @fatherlinux

carlwgeorge commented 2 years ago

I'm curious, would CentOS Stream 9 be an option here? It's default gcc is 11 (no need to enable an alternative toolset), it has all four architectures discussed here (aarch64, ppc64le, s390x, and x86_64), and it's maintained until approximately 2027-05-31 (depends on the exact date of the end of RHEL 9 full support phase).

mattip commented 2 years ago

would CentOS Stream 9 be an option here

What glibc does it pin?

mayeut commented 2 years ago

We plan to add s390x this year

Good to know, I posted both on centos-devel mailing list & https://discussion.fedoraproject.org/t/any-plans-to-support-s390x-on-centos-stream-8/36857/4, I now have the answer, thanks.

but if it would make UBI the preferred choice for this

I guess it would EDIT: Just tested if anything else would be missing: Error: Unable to find a match: bison gcc-toolset-11-binutils gcc-toolset-11-gcc gcc-toolset-11-gcc-c++ gcc-toolset-11-gcc-gfortran readline-devel tk-devel gdbm-devel libpcap-devel libidn-devel uuid-devel libXext-devel libXrender-devel mesa-libGL-devel libICE-devel libSM-devel As was said on the discourse thread, UBI8 is probably meant to redistribute runtime containers. As such, there's a bunch of devel packages missing (in addition to gcc-toolset-11). Even if we get those packages in for the manylinux image, a project downstream will probably end-up missing foo-devel and report here where we'd ask them to follow UBI FAQ#35. It does not seem sustainable.

I'm curious, would CentOS Stream 9 be an option here?

No, manylinux_2_28 stands for glibc 2.28 and assumes a bit more about a bunch of other versions, mostly GCC, GLIBCXX symbol versions that gets linked in the binary. In this regard, using gcc-toolset-11 on CentOS Stream 8 is different than using the default gcc 11 on CentOS Stream 9. It might be a viable option for a future manylinux_2_34 though.

carlwgeorge commented 2 years ago

Ah, I was wondering what the name was derived from. C9S indeed has glibc 2.34.

messense commented 2 years ago

I guess it would make sense to add https://github.com/PyO3/maturin the list of tools in the description ?

Sure.

h-vetinari commented 2 years ago
base image EOL gcc-toolset-11 aarch64 ppc64le s390x bug fix availability
CentOS Stream 8 2024-05-31 yes yes yes planned 1st
UBI 8 2029-05-31 NO yes yes yes 2nd
AlmaLinux 8 2029-05-31 yes yes soon planned 3rd
RockyLinux 8 2029-05-31 yes yes planned planned 3rd

Looking at the table @mayeut edited into the OP (thanks!), sans footnotes, IMO the clear winner would be UBI-with-gcc-toolset-11. Without that, it'd be much less clear IMO (perhaps C8S first and then switch to Alma when C8S goes EOL?).

I can't make any promises, but if it would make UBI the preferred choice for this, I suspect it would be viewed quite favorably. cc: @fatherlinux

I once had a wish fulfilled by @fatherlinux, can confirm that nice surprises can (be made to) happen. ;-)

kpfleming commented 2 years ago

I'm not clear on why 'bug fix availability' for UBI8 is listed as '2nd'. It is handled identically to C8S, with packages released on the same schedule.

fatherlinux commented 2 years ago

Technically, features would arrive first on CS8 but some security bugs are fixed faster in UBI8 so it depends on whether you want features or security.

I'll look into that veraion of gcc-toolkit :-)

Best Regards Scott M

On Mon, Feb 14, 2022, 6:09 AM Kevin P. Fleming @.***> wrote:

I'm not clear on why 'bug fix availability' for UBI8 is listed as '2nd'. It is handled identically to C8S, with packages released on the same schedule.

— Reply to this email directly, view it on GitHub https://github.com/pypa/manylinux/issues/1282#issuecomment-1038955074, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA4WZP6FQEUVAKEIP6PJTDU3DPIJANCNFSM5OCHJJ3A . You are receiving this because you were mentioned.Message ID: @.***>

carlwgeorge commented 2 years ago

It is handled identically to C8S, with packages released on the same schedule.

They're very much not on the same schedule. UBI is on the same update schedule as RHEL, with most updates being batched up into minor releases, with a small number of updates in between. Updates in CentOS Stream are released as they pass QA. C8S already has many fixes that RHEL8/UBI8 won't get until 8.6 comes out in a few months. As Scott mentioned, some select security fixes get prioritized to be released in RHEL first, but then are released to CentOS Stream shortly after. The vast majority of updates (including most security fixes) are in CentOS Stream first, usually months ahead of RHEL and of course the clones.

jaboutboul commented 2 years ago

hello everyone. just to add in 2 cents here from the AlmaLinux side of things. We currently have ppc64le in beta (https://repo.almalinux.org/almalinux/8.5/isos/ppc64le/) and plan on the release next week. For s390x we are getting there and we should have that by the end of the month or at the latest early next month.

cwiede commented 2 years ago

Hello - just replying to the question here: https://github.com/pypa/manylinux/issues/1249#issuecomment-1037152990

Thanks for your efforts. Indeed everything seems to work for me with mayeut/manylinux:manylinux_2_28_x86_64 using PySide6. Side note: I'm using another linux to generate the sources using shiboken6 which didn't work out of the box on this image, but this is fine (it has also been the case with PySide2 / manlinux2014). The compilation itself did work on the image.

mayeut commented 2 years ago

Side note: I'm using another linux to generate the sources using shiboken6 which didn't work out of the box on this image

Do you have a link to a reproducer for this ? Any idea what's going wrong ?

cwiede commented 2 years ago

Do you have a link to a reproducer for this ? Any idea what's going wrong ?

I think that's just because of some crazy dependencies of shiboken6_generator. It might even be possible to get that working by installing more centos packages, but since the workaround was already in place for manylinux2014 I didn't bother too much.

EwoutH commented 2 years ago

I would love an manylinux_2_28 image, is there any update on this effort?

duburcqa commented 2 years ago

The documentation on the main page is wrong. It is referring the image url for manylinux_2_24 where it is supposed to be manylinux_2_28.

mayeut commented 2 years ago

The documentation on the main page is wrong. It is referring the image url for manylinux_2_24 where it is supposed to be manylinux_2_28.

This is now fixed.

messense commented 2 years ago

quay.io/pypa/manylinux_2_28_ppc64le this repository is empty.

EwoutH commented 2 years ago

Are there any issues with the manylinux_2_28_ppc64le image, seeing that the repo is empty?

mayeut commented 2 years ago

there are issues with travis-ci on ppc64le at the moment, all ppc64le images are impacted, not only manylinux_2_28_ppc64le. No ETA on when this will be fixed.

EwoutH commented 2 years ago

Thanks for the update!

EwoutH commented 2 years ago

Took keep momentum, would it be time to start talking about the next manylinux version? Would that be manylinux_2_34, using glibc 2.34, based on a version 9 BaseOS? (maybe AlmaLinux 9?)

tiran commented 2 years ago

Took keep momentum, would it be time to start talking about the next manylinux version? Would that be manylinux_2_34, using glibc 2.34, based on a version 9 BaseOS? (maybe AlmaLinux 9?)

It's a bit premature to start with manylinux_2_34. The wheel would not work on latest Debian Stable or Ubuntu 20.04. Their glibc versions are lower than 2.34. By the way there is a realistic chance that Red Hat will provide support to create a manylinux container based on RHEL UBI.

EwoutH commented 2 years ago

The wheel would not work on latest Debian Stable or Ubuntu 20.04. Their glibc versions are lower than 2.34.

For those OS there is the brand new manylinux_2_28 image right? Currently multiple popular releases like Fedora 35 and 36 and Ubuntu 21.10 and 22.04 LTS already support glibc 2.34+. Also many rolling releases like Gentoo, Manjaro Stable and openSUSE Tumbleweed support it.

The whole philosophy behind PEP 600 was that it could speedup new manylinux version:

And second, every time we move manylinux forward to a newer range of supported platforms, or add support for a new architecture, we have to go through a fairly elaborate process: writing a new PEP, updating the PyPI and pip codebases to recognize the new tag, waiting for the new pip to percolate to users, etc. None of this happens on Windows/macOS; it’s only a tax on Linux maintainers. This slows deployment of new manylinux versions, and consumes part of our community’s limited PEP review bandwidth, thus slowing progress of the Python packaging ecosystem as a whole. This is especially problematic for less-popular architectures, who have less volunteer resources to overcome these barriers.

Also consider that packages can choose which manylinux images they want to build. A popular distribution that cares about performance can build a lot, while smaller packages can choose a few that they see fit.

Especially for Ubuntu 22.04 LTS a new manylinux image could be very useful. That could either be glibc 2.34 or 2.35, but I think that considering we need a base image 2.34 might be easier.

mayeut commented 2 years ago

It's a bit premature to start with manylinux_2_34. The wheel would not work on latest Debian Stable or Ubuntu 20.04. Their glibc versions are lower than 2.34.

Agreed

By the way there is a realistic chance that Red Hat will provide support to create a manylinux container based on RHEL UBI.

Good to know !

Also consider that packages can choose which manylinux images they want to build. A popular distribution that cares about performance can build a lot, while smaller packages can choose a few that they see fit.

You probably won't get better performance since you'll probably end up using the same toolchain thanks to RH effort to have gcc-toolset-* on RHEL8. You'll mostly get higher complexity with no added benefits.

carlwgeorge commented 2 years ago

By the way there is a realistic chance that Red Hat will provide support to create a manylinux container based on RHEL UBI.

I think for manylinux_2_34, UBI 9 is already a great option. It has glibc 2.34 and gcc 11 (as the default, no need for scl enable).

mayeut commented 2 years ago

quay.io/pypa/manylinux_2_28_ppc64le this repository is empty.

there are issues with travis-ci on ppc64le at the moment, all ppc64le images are impacted, not only manylinux_2_28_ppc64le.

ppc64le images have been published.

cc: @messense

bryanculver commented 2 years ago

I noticed when spinning up a manylinux_2_28 container that there is no longer a "default" Python executable (not found in PATH).

Obviously for building it's expected to go to the specific version in /opt/python/ but what (if there is) is the preferred/recommended way to add Python into the PATH, say if we have a helper script that's written in Python?