search

indygreg / python-build-standalone

Produce redistributable builds of Python
BSD 3-Clause "New" or "Revised" License
1.71k stars 107 forks source link

Building with OpenSSL 3.0 for the GitHub Release binaries #175

Closed Edward-Knight closed 9 months ago

Edward-Knight commented 1 year ago

OpenSSL 1.1.1 is going EOL in September, and it looks like the builds here are currently using 1.1.1s. From a quick glance it seems like CPython supports building with OpenSSL 3.0 from 3.9 onwards.

I was wondering if there were any plans about upgrading? I imagine Python 3.8 will still need to use OpenSSL 1.1.1, but I believe the other builds can be upgraded. I can help with an MR if there isn't anything blocking.

indygreg commented 1 year ago

We should ideally migrate to OpenSSL 3 for every CPython version that supports it. This has been on my mind for a while I just haven't done it yet.

On Windows, we might want to maintain parity with whatever CPython is shipping. i.e. we'll want to wait on upstream to ship a CPython 3.9 using OpenSSL before we do. (I anticipate CPython will run into issues getting OpenSSL 3 working right on Windows because pretty much nobody builds CPython from source on Windows and the build configurations that aren't identical to the official CPython releases aren't well tested.) But if CPython does have OpenSSL 3 test coverage, by all means upgrade.

Patches would be very much appreciated!

Edward-Knight commented 1 year ago

From poking around in the CPython GitHub Actions build.yml and PCBuild/get_externals.bat it seems that they currently build using these versions of OpenSSL:

Linux SSL tests Linux build macOS build macOS installer Windows build
Python 3.8 1.0.2u, 1.1.1u, 3.0.9, 3.1.1 1.1.1u 1.1 (PR for 3.0) 1.1.1u 1.1.1u
Python 3.9 1.0.2u, 1.1.0l, 1.1.1u, 3.0.9, 3.1.1 1.1.1u 1.1 (PR for 3.0) 1.1.1u 1.1.1u
Python 3.10 1.1.1u, 3.0.9, 3.1.1 1.1.1u 1.1 (PR for 3.0) 1.1.1u 1.1.1u
Python 3.11 1.1.1u, 3.0.9, 3.1.1 1.1.1u 3.0 1.1.1u 1.1.1u

It looks like OpenSSL 3.0 should work on Python 3.8. The ticket I linked in my original post does have some PRs into 3.8, and although OpenSSL 3.0 tests for 3.8 were removed in https://github.com/python/cpython/commit/438f8cd77d5d532352b6dc02e2cd6457ea76cc89, it looks like they were added back in in https://github.com/python/cpython/commit/ddd495e0552be409e3c8c280a6f7a69ed2e6425e.

@indygreg would you be okay with a PR that:

erlend-aasland commented 1 year ago

FTR / FWIW, there are three open CPython PRs for running the macOS CI with OpenSSL 3.0:

(These PRs are awaiting approval from the respective release managers, since merging to security fix branches is constrained.)

The macOS CI for 3.11, 3.12 and 3.13 (main) are already using OpenSSL 3.0. The devguide also recommends OpenSSL 3.0.

See also:

Quoting https://github.com/python/cpython/issues/99079#issuecomment-1581927371:

If we assume that that we should focus on 3.0.x support on macOS for the time being at least, I think the first step would be to update our GitHub Workflow macOS CI jobs for 3.11+ to use 3.0.x from Homebrew: currently they are using 1.1.x. We should also update the devguide recommendations for macOS to use 3.0.x; there is a https://github.com/python/devguide/pull/1058 that should be changed to use 3.0.x. Also, any macOS buildbots should be updated, as feasible. With that additional experience we can then move the macOS installers to using 3.0.x sometime in the not-too-distant future.

erlend-aasland commented 1 year ago

Also, the macOS installer uses the following:

https://github.com/python/cpython/blob/04492cbc9aa45ac2c12d22083c406a0364c39f5b/Mac/BuildScript/build-installer.py#L243-L256

Edward-Knight commented 1 year ago

Thanks Erlend I'll update my table to include that

Edward-Knight commented 1 year ago

OpenSSL has changed license in 3.0 to Apache License 2.0, which I assume isn't an issue but I'll note it here as it will require an update to the license info we bundle: https://www.openssl.org/source/license.html

erlend-aasland commented 11 months ago

FYI, OpenSSL 3.0.x is now used in the official binary builds: https://github.com/python/cpython/issues/99079#issuecomment-1657824684

Edward-Knight commented 10 months ago

Thanks Erlend 👍

indygreg commented 9 months ago

I think I'm pretty close to having OpenSSL 3.0 working. Just need to figure out how to handle a new dependency on libatomic on Linux.

On Linux and macOS I should be able to get OpenSSL 3.0 working across the board.

On Windows, I've only got OpenSSL 3.0 working on CPython 3.11. It looks like the latest CPython releases only support OpenSSL 3.0 in the Visual Studio build files on 3.11+. I'm guessing upstream will be backporting OpenSSL 3.0 Windows support to older CPython since OpenSSL 1.1 is EOL and the community will likely throw a fit if CPython ships OpenSSL 1.1 with known security vulnerabilities (once an unpatched vuln in 1.1 inevitably drops). I'm content waiting for upstream to provide official OpenSSL 3.x support before shipping it in python-build-standalone.

If someone knows what CPython is actually planning for OpenSSL 3 on Windows, please comment!

erlend-aasland commented 9 months ago

If someone knows what CPython is actually planning for OpenSSL 3 on Windows, please comment!

Official CPython binaries from python.org are now using OpenSSL 3 for Windows builds (for the main, 3.12, and 3.11 branches).

indygreg commented 9 months ago

Official CPython binaries from python.org are now using OpenSSL 3 for Windows builds (for the main, 3.12, and 3.11 branches).

I realize this.

But CPython <3.11 are still in support. And OpenSSL 1.1 is out of support. So what's CPython's plans for OpenSSL 3.x on <3.11? Or put another way, what is CPython going to do when an OpenSSL 1.1 vulnerability drops? Are CPython <3.11 going to remain exposed? Is CPython going to distribute a patched OpenSSL 1.1?

erlend-aasland commented 9 months ago

Quoting https://github.com/python/cpython/issues/99079#issuecomment-1694100406:

None: binaries are no longer produced for these versions, and the code is already compatible (enough).

I think that answers most of your questions.

Edward-Knight commented 9 months ago

@indygreg I've pushed my WIP code to #189 - hopefully it will be useful to you. I've been working on it on-and-off in my fork for a while but haven't managed to get all the architectures working unfortunately