search

pantsbuild / pants

The Pants Build System
https://www.pantsbuild.org
Apache License 2.0
3.34k stars 638 forks source link

update the manylinux wheel glibc version to 2.28 (#21650) #21660

Closed benjyw closed 3 days ago

benjyw commented 3 days ago

While trying to land #21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like /__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node). This appears to be the same symptoms as https://github.com/actions/checkout/issues/1809 which pointed the the deprecation notice at https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/. If this all sounds familiar that is because we mostly cleaned this up in #21133, but kept a fewer uses of the older actions for manylinux2014 compatibility. However, from the notice:

"To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true as an ‘env’ in their workflow or as an environment variable on your runner machine. This will only work until we upgrade the runner removing Node16 later in the spring. (emphasis added)"

From the wheel job failures during #21528 and my attempts to fiddle with all of ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION, ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION, ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something.

As a consequence the version of both actions/upload-artifacts and actions/checkout is bumped to v4. To make this testable now and in the future, wheels are now build on ci config changes.

Why is bumping to manylinux_2_28 (the next oldest manylinux) not a big deal? The "2.28" refers to glibc 2.28 in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution.

NOTE: I think we should bump the manylinux version in main regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per #21616.

ref #21195 #21616

benjyw commented 3 days ago

Yep. Since you've done the deep dive here, could you phrase that footnote?

cburroughs commented 3 days ago

"Due to deprecations by GitHub, the minimum supported glibc version for Pants wheels is now 2.28. Unless you are running Pants with in-repo plugins on a Linux distro from before 2018, this should have no effect."

huonw commented 3 days ago

I think this could applies to more than just the pants wheels/in-repo plugins, in that the new code could call some glibc 2.28 APIs, and thus fail to run with older versions. I don't know if this occurs in practice, though. (The code calling these APIs could either be direct Pants Rust code (since it's now being built-with/linked-against newer glibc), or a transitive dependency that's itself able to start using newer wheels).

Is it possible to get 2.23.0 out, as is, without this change? And then cherry pick for 2.23.1+ (it's a bit of a weird change to happen in the middle of a release cycle, but our hands are kinda forced...)

benjyw commented 3 days ago

Is it possible to get 2.23.0 out, as is, without this change? And then cherry pick for 2.23.1+ (it's a bit of a weird change to happen in the middle of a release cycle, but our hands are kinda forced...)

I don't think so. The wheel building shards just fail.

benjyw commented 3 days ago

Is it possible to get 2.23.0 out, as is, without this change? And then cherry pick for 2.23.1+ (it's a bit of a weird change to happen in the middle of a release cycle, but our hands are kinda forced...)

I don't think so. The wheel building shards just fail.

But I am open to ideas.

huonw commented 3 days ago

I imagine we'll need to cherry-pick this to the 2.22.x branch too?

benjyw commented 3 days ago

Yeah, and 2.24.x...

cburroughs commented 2 days ago

There is probably some option where -- with much hand-waving -- we use docker environments and https://cibuildwheel.pypa.io/en/stable/, but I think that falls in the unreasonable LoE bucket.

I think this could applies to more than just the pants wheels/in-repo plugins, in that the new code could call some glibc 2.28 APIs, and thus fail to run with older versions. I don't know if this occurs in practice, though. (The code calling these APIs could either be direct Pants Rust code (since it's now being built-with/linked-against newer glibc),

So I have not done more than a toy examples with Rust, but thinking about this more we build the Pants Pex itself on ubuntu22, so absent some cross-compiling Rust magic I'd expect the minimum glibc version that works with would already be 2.35. If that is right, the only impacted workflows would be ones that use the Pants released wheels but not the Pants released Pex. I'm not sure there are any?

huonw commented 2 days ago

I think it's slightly more subtle than that, since it depends it on which wheels are selected: here's what's in 2.22.1's PEX-INFO:

{
    "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "8c0396211a76d27f152910bbbfdcabe20f4e03be0e4f54aca9c3bd79414bf3df",
    "ansicolors-1.1.8-py2.py3-none-any.whl": "18687c11ecd3f2ee80dffbcbfd248ea50309fa65a2df1d805c6bf1cec8b1f33d",
    "chevron-0.14.0-py3-none-any.whl": "d7316200cba6fc6d3935ba3e8f787b39e0a2d6bc64f091501285aba47a41845f",
    "fasteners-0.16.3-py2.py3-none-any.whl": "ccabc462ef0f9c6905cdf1c3c3709fca2f79db97310df0a71f29b1771bb48539",
    "ijson-3.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "0326508a88314ae80cddf17bb354e72bf415f75bd544635854e16b4018e3dd4a",
    "node_semver-0.9.0-py3-none-any.whl": "333d5510aeb120cfb3abdf9404720c9f23b7e4c7c78f11d761647da102c9c6fb",
    "packaging-21.3-py3-none-any.whl": "264b338d5e85b5a986fc45f766b53065dcaccd7cb35d204ff319bd80c503c5f6",
    "pantsbuild.pants-2.22.1-cp39-cp39-linux_x86_64.whl": "34c467e7f0f1124884b1e2ac263401a3088b674c66fd5eae147bc3f751e086ba",
    "pex-2.3.1-py2.py3-none-any.whl": "8d43c1ed16877f3d2ed39e5ef3e92aa8885d4427cd7d9ce62683c922a5c0cd07",
    "psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "b40c0935c1f080f868ad92f09fe97d9528ac7a99fdf740550db21365709b83ff",
    "pyparsing-3.1.2-py3-none-any.whl": "b422fdeeec20dcd8540113cbf5f871361708d7a98a86fc2152aef1484440d9aa",
    "python_lsp_jsonrpc-1.0.0-py3-none-any.whl": "c048d3fe8132ce68f303b1e60d99be11cfe8d7d12181ac982309689a9589cedb",
    "setproctitle-1.3.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "5c538df39985111bc0ec3d0deccf5ef3a3de9268137f6271bded8f3d672322d5",
    "setuptools-63.4.3-py3-none-any.whl": "4e4fc62fd129faff0b16783574fe9b0c356bf52c1cc6217e61c96a6afb14bbca",
    "six-1.16.0-py2.py3-none-any.whl": "a8dffba69ff16231827d7133867794d7e85295e22c36fbdc496b3d80d6d8d2cc",
    "toml-0.10.2-py2.py3-none-any.whl": "5fb6845639412b75ccc50cbb2cb70716517b59e3cbc6b832b1b652bb5e24bb99",
    "types_PyYAML-6.0.3-py3-none-any.whl": "3fa6b44036ae62cd583d37db411a9a1b1ebe01dc45dd64305cf3e2c2c338fd3e",
    "types_setuptools-62.6.1-py3-none-any.whl": "6d89dff70d523e7df3a9cfa1cb9f103bab6c976f03ab55a0719d361a4ba000e3",
    "types_toml-0.10.8-py3-none-any.whl": "46e8d165f4bd8790d3b526580067cfc1169f98c92a178b8c6ffcd7f6a66aca71",
    "typing_extensions-4.3.0-py3-none-any.whl": "2370627dba47a337f8ecc3207bc70ef64bb02f556b82d78e0d6112982ebbd3c1",
    "ujson-5.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "f54f9fa90a80781314b55ca217721f3be8ce41424c3fdbce3e5792876cf40d28"
}

So all of those wheels are manylinux2014 or older, other than the Pants one.