search

python / cpython

The Python programming language
https://www.python.org
Other
63.37k stars 30.35k forks source link

Update JIT compilation to use LLVM 19 #123714

Closed savannahostrowski closed 1 week ago

savannahostrowski commented 2 months ago

The JIT currently uses LLVM 18. The final release of LLVM 19 is slated for September 17th. We should update to the latest version.

Linked PRs

savannahostrowski commented 2 months ago

I'm going to start poking around at this.

ZeroIntensity commented 2 months ago

I'm a little concerned about this. I was on a laptop a few weeks ago that had LLVM 17 installed, and I upgraded it to 18 to build the JIT -- it completely broke my system, and took several hours of debugging in a recovery shell to find the problem and fix it. Will we require LLVM 19, or just support it?

brandtbucher commented 2 months ago

Probably require, especially since the feature is optional and (still) experimental. We're also going to want to use some new features of 19, since it means we can get rid of some ugly hacks and will probably pave the way for easier performance improvements on our end (so it's not just updating our toolchain for the sake of updating).

BTW, multiple LLVM versions can typically be installed side-by-side, and our build process works correctly even if clang is an incompatible version but clang-18 exists (for example). Did just installing the new version break everything, or did you completely replace the existing 17 installation?

We should also hold off on this until LLVM 19 is available for all of the platforms we care about. IIRC, Homebrew support was way behind in the 18 cycle.

brandtbucher commented 2 months ago

Also, if your machine was macOS, installing an incompatible LLVM on your PATH will probably break stuff (which is why Homebrew doesn't do it by default). The build scripts know how to find a Homebrew install that isn't on the user's PATH though.

ZeroIntensity commented 2 months ago

Did just installing the new version break everything, or did you completely replace the existing 17 installation?

I'm not totally sure (I don't know much about package managers), probably the latter. FWIW, the machine was Arch.

savannahostrowski commented 2 months ago

@brandtbucher Yeah, I wouldn't want to merge the PR in until the installers catch up. It'll make the dev experience a lot more palatable than building LLVM from source 😅. I just kind of figured I'd get a head start trying to wrap my head around it since rc-4 should be pretty stable.

picnixz commented 2 months ago

I'm not totally sure (I don't know much about package managers), probably the latter. FWIW, the machine was Arch.

Assuming that Arch is Arch Linux for you, IIRC, it does not support having two different versions of LLVMs living together (but I don't know if it's still the case and if you can just build it from source and isolate it). On the other hand, upgrading to LLVM 18 can break some of your system packages (I remember that there were some issues with LLVM 16 which broke Steam games, but I'm not surprised that it can break a rolling-release).

ZeroIntensity commented 2 months ago

it does not support having two different versions of LLVMs living together

Yeah, I learned that the hard way. I'm not opposed to upgrading to LLVM 19 for the JIT, I just think we should keep compatibility in mind.

brandtbucher commented 2 months ago

Hm, okay, definitely something to keep in mind then (though as a general improvement for the future, I don’t see it as a blocker for this particular issue).

I wonder if we can do something like we do for Homebrew, where there’s some sort of standard install location that’s not on the user’s PATH. Another option could be to let a user pass a path to their LLVM installation via a configure arg instead of us trying to find it. Or we could attempt to support a range of versions once the JIT is stable, which could work but seems like it has the potential to massively increase the complexity of the build scripts (and our testing matrix).

See also GH-115869 for related discussion about removing the LLVM dependency.

picnixz commented 2 months ago

I wonder if we can do something like we do for Homebrew, where there’s some sort of standard install location that’s not on the user’s PATH

Should probably be possible according to https://llvm.org/docs/GettingStarted.html#getting-the-source-code-and-building-llvm. For other non-rolling release distributions, multiple versions can co-exist together (I have multiple LLVM versions on openSUSE Leap).

theoparis commented 1 month ago

Can this be backported to 3.13 in time for the release? I am using Gentoo Linux and python 3.14/main isn't an option unless I patch the majority of gentoo's packages to support it. I have already updated to llvm 19 so if it can't be backported I'll have to disable the jit option for now.

ZeroIntensity commented 1 month ago

I don't think we're going to even be merging this before the 3.13 release -- your best bet is to disable the JIT (it's experimental, anyway).