macOS linker warnings in macOS ventura

[pablogsal]

Turns out ld shipped with Xcode 14+ emits a warning when using -undefined dynamic_lookup.`

ld: warning: -undefined dynamic_lookup may not work with chained fixups

Some investigation reveals that in fact -undefined dynamic_lookup doesn't work when:

• Link a shared library with the option
• Link it with a program that uses the chained-fixup introduced from macOS 12 and iOS 15

This is because -undefined dynamic_lookup uses lazy-bindings and they won't be bound while dyld fixes-up by traversing chained-fixup info.

However, we build extension modules with as bundles (-bundle) and they are loaded only through dlopen, so it's safe to use -undefined dynamic_lookup in theory. So the warning produced by ld64 is likely to be a false-positive.

ld64 also provides the -bundle_loader <executable> option, which allows resolving symbols defined in the executable symbol table while linking. It behaves almost the same with -undefined dynamic_lookup, but it makes the following changes:

• Require that unresolved symbols among input objects must be defined in the executable.
• Lazy symbol binding will lookup only the symbol table of the bundle loader executable. (-undefined dynamic_lookup lookups all symbol tables as flat namespace)

See "New Features" subsection under "Linking" section for chained fixup developer.apple.com/documentation/xcode-release-notes/xcode-13-release-notes for more information.

Also, check the same problem in ruby where most of this information is taken from.

[wjakob]

We can create it and package it (and use when building the interpreter) but extensions need to pick it up using whatever build system they are using, no?

That was the reason for packaging it up in my binding project. Of course, the implication wasn't that CPython should rely on this external link.

Probably the most portable way would be if the Python binary itself could be queried to list its own exported symbols.

[ronaldoussoren]

However, I imagine we will likely want to create this symbol list ourselves "just-in-time" in our build tooling to future-proof it without creating an external dependency. 3.12 is still in flux so the list might change, and 3.13 is around the corner.

We can create it and package it (and use when building the interpreter) but extensions need to pick it up using whatever build system they are using, no?

With some luck most build systems would pick up flags from python3-config or the equivalent information in sysconfig.

103306 contains a related discussion about using -U flags, although as only one of the options to drop -undefined dynamic_lookup to improve the build experience (e.g. no silent failure due to missing symbols). I'll close that other issue because this issue is a lot clearer about the way forward.

[ronaldoussoren]

Generating the file should be easy enough, e.g.:

libpython.sym: $(LDLIBRARY)
     nm $(LDLIBRARY) | grep ' T ' | awk '{ nm = substr($3, 2); print("-U ", nm) }'  > libpython.sym

I have not yet tried to actually use this though, let alone thought about the correct way to integrate this into the installation and build system.

This will list more symbols than are technically part of the API, but keeps us closer to the current linking behaviour w.r.t. using private but exported symbols. That's not ideal, but does avoid bug reports about link errors on macOS that don't happen on other unix-y platforms.

[Dave-Allured]

I submitted an upstream patch to GNU libtool, to add -Wl,-no_fixup_chains for Mac builds. If accepted, this will hopefully propagate to many packages using libtool, and mitigate that lookup warning. I welcome any feedback, here or on that GNU ticket.

This would mean that packages wanting to use chained fixups would need to override this new libtool default.

[Dave-Allured]

Sorry, I mean to include that link: https://savannah.gnu.org/support/?111069