indygreg
commented
2 years ago I'll look at the diff later, but my recollection is CPython didn't do weak linking properly until a 3.9 point release. I believe things were finally implemented somewhat properly when Apple M1s became a thing and they were forced to use the macOS 11.0 SDK to target M1 and this exposed them to portability issues when x86-64 binaries were built with the 11.0 SDK.
It took them a while to do weak linking properly, as there were definitely random symbols not doing it properly. And my recollection is some initial attempts at introducing weak linking didn't actually work properly. I think that's why there is code in this repo for force disabling specific symbols.
The code to support weak linking is scattered all over the CPython code base and much of it was backported. Since much of it was all tied up in M1 support and 3.8 didn't get those backports until much later, I wouldn't at all be surprised if 3.8 has bugs in this area.
As for this project, I very much dislike the fact that we prevent the use of some new symbols instead of using weak linking. This does have performance implications and I'd love to see us do the right thing.
A thing I struggle with is automated enforcement of symbol visibility. The Rust code in this repo for verifying some binary properties does detect some banned symbols. But it doesn't currently recognize - and allow - weakly linked symbols. I'd love to change this. I suspect there's even room to parse .tbd files in macOS SDKs to verify symbol availability and require weak linking when appropriate. I even wrote a Rust crate (https://crates.io/crates/text-stub-library) for parsing .tbd files with this potential goal in mind.
Another issue here is presence of weak symbols impacts downstream projects linking the raw object files. We might need to annotate weak symbols or other linker metadata in the PYTHON.json file so weak symbols can be handled accordingly. Otherwise, I believe weak symbols can get promoted when linking the object files. So if we introduce weak symbols, we need to be sure PyOxidizer handles them correctly.
ddeville
Python (at least 3.8) has trouble dealing with weak symbols that are defined in macos availability macros and ends up linking them strongly which causes it to fail launching on older OSes since it can't find the symbols at runtime. I've noticed issues on 10.10 and 10.11 due to symbols added in 10.12 (mostly
getentropy) but this PR attempts to deal with this in a slightly more generic manner.I know that Python has been attempting to fix this and maybe 3.9 doesn't have that issue but I haven't looked too closely.