macOS builds linking against gettext in /usr/local/opt

[efroemling]

I was rebuilding everything today in the 3.7 branch to pick up latest changes and although all Python builds completed successfully I noticed that my App builds then started failing with a lot of 'Undefined symbol: _libintl_xxxx' errors.

I then rebuilt using the commit that I last used (3.7 commit bf1ed73d0d5ff46862ba69dd5eb2ffaeff6f19b6 from January) and got the same errors, so it doesn't seem related to the recent commits here. This is only happening on the Mac builds; iOS and tvOS seem to be fine. This is with macOS 10.15.5 and the Xcode 11.5 update from the other day.

Looking through build logs, it seems that Python config is finding libintl.h which is part of GNU gettext I believe and for me only exists in my homebrew setup in /usr/local. I'm also seeing lots of -I/usr/local/include in compile commands. I don't have any CFLAGS/etc stuff set as far as I'm aware.

If I run nm build/macOS/libpython3.7m.a | grep -I libintl I see quite a few undefined symbols mentioned, but doing the same thing on the one I built in January comes up empty.

This seems to not just be my builds; If I grab the prebuilt macOS binary from https://briefcase-support.org/python?platform=macOS&version=3.7 and run otool -L bin/python3, I see:

    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1673.126.0)
    /usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.7.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
    /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
    /usr/local/opt/xz/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.5.0)
    /usr/lib/libsqlite3.dylib (compatibility version 9.0.0, current version 308.4.0)
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)

And if I temporarily rename /usr/local/Cellar, that binary fails to launch.

Any idea what would have changed here as compared to a few months ago? These Python builds shouldn't be looking at or depending on anything under /usr/local, right?..

[freakboy3742]

Thanks for the report. You're completely correct that we shouldn't be depending on anything in /usr/local, and the binaries should be completely self contained, statically linked binaries.

It's interesting that liblzma is the list you've found in the official list; XZ is compiled as part of the support build process, and it should be that copy that is being linked - which suggests that there's some leakage in the configuration of the static modules. I made some changes to the build system for the most recent packages (3.7-b3), so it's possible I have messed up there.

This also suggests that we should add that otool check as a validation to the build process.

[efroemling]

A bit of info I've found while digging: I've got a CI server set to run these Python builds every so often since I have a few customization that I want to make sure don't break. I was looking through old build logs and found that sometime between March 30th and April 25th Python's configure began detecting gettext stuff in /usr/local.

Here is the complete build log from March 30th: https://www.dropbox.com/s/agc2x0sf6o4gxsa/build_log_2_months_ago.txt?dl=0

And here is from April 25th: https://www.dropbox.com/s/ysenjhkrm99icck/build_log_1_month_ago.txt?dl=0

gettext seems to be used by a bunch of homebrew stuff so I'm sure it was there before March 30th. However seeing links like this make it sound like it at some point was 'keg-only' and not in default paths. If I do 'which gettext' I see '/usr/local/bin/gettext' so I'm wondering if that changed sometime between March 30th and April 25th.. 🤔. Do you see the same?

[freakboy3742]

The commit logs for the 3.7 branch don't show any changes between January 19 and last weekend (May 31) - and that's consistent with my own memory and timesheets. We've only recently added CI to our own builds, so I unfortunately can't confirm any historical build logs. I did some work on the 3.8 branch in early May, but that work wasn't committed until it became a part of last weekend's effort.

Your hypothesis about gettext recently becoming non-keg-only definitely sounds plausible, though.

It sounds like it might be advisable for us to remove /usr/local from $PATH as part of the build script, to avoid any possibility of homebrew tooling being picked up.

[freakboy3742]

This has now been resolved (albeit with a hack that working CI, but won't work locally - uninstalling gettext from the CI machine). A fix that removes /usr/local from the macOS build path would be preferable, but /usr/local appears to be hard-coded as part of the macOS build tooling.

[efroemling]

Thanks for the followup. I'm doing basically the same hacky thing in my own build scripts (temporarily uninstalling homebrew's gettext when building). In positive news, according to this it sounds like the plan for homebrew in the future is to only support installs at /opt/homebrew instead of /usr/local, so hopefully that'll make it easier to keep builds paths clean when desired.