Linux shared object builds erroneously export 3rd party locally used symbols

[ceztko]

I noticed linux .so builds export tons of 3rd party symbols, such as openssl, tk, and others. This can be verified by downloading one official build, such as[1], and run the following command:

nm -D install/lib/libpython3.9.so

This produces the following output: standalone-libpython3.9-nm.txt . Here is an excerpt:

00000000008d0820 T X509_supported_extension
00000000008ba1f0 T X509_time_adj
00000000008ba300 T X509_time_adj_ex
0000000000970f00 T X509_to_X509_REQ
0000000000971840 T X509_TRUST_add
[...]
0000000000419860 T PyObject_SetAttrString
0000000000350640 T PyObject_SetItem
00000000003500b0 T PyObject_Size
0000000000371790 T PyObject_Str
00000000003ecd60 T PyObject_Type
00000000003fdf00 T PyObject_Vectorc

As you can see licit Python API symbols are exported together with openssl symbols. Look by comparison the output produced by the same command on a libpython3.8.so as packaged in a ubuntu system: ubuntu-libpython3.8-nm.txt . Here mostly official python API symbols are exported, and no trace of openssl/tk ones.

The behavior as shown in python standalone builds is problematic as this may cause software dynamically loading symbols to erroneously bind to the standalone libypthon exported ones, incurring in ABI incompatibilities and crashes, making your builds unusable for me. To play fair, the standalone libpython3.9.so should export the minimal set of needed symbols, such as the python API ones, and not export any of the 3rd party symbols that are used locally inside it. It's strange this is not just happening in your builds, since at some point you probably rely on the official python build infrastructure which I believe it should already take care of this. Nevertheless this is usually achieved in different ways:

1) You set the compiler to not export symbols by default. This is the default in MSVC compilers but not on gcc/clang. On the latter, this is achieved by setting -fvisibility=hidden on the compiler options. Then selectively all the public symbols are exported: this is done by decorating the symbols with__attribute__ ((visibility ("default"))). The infrastructure to selectively export only public API symbols seems to be already set up in the official python source, see the expansion of the PyAPI_FUNC and Py_EXPORTED_SYMBOL macros; 2) You select a list of static libs to not export any symbol from, or just all of them, using the linker options -Wl,--exclude-libs,ALL[2]. This linker options is not available in the apple linker, so I don't recommend this strategy.

I ask you support where I could experiment enforcing the -fvisibility=hidden compilation flag when the final libpython3.9.so is built, which I believe it could just the fix for the problem I am observing, and apply the same on your tree when this verified to produce a working binary that is exporting only the required symbols.

[1] https://github.com/indygreg/python-build-standalone/releases/download/20211017/cpython-3.9.7-x86_64-unknown-linux-gnu-lto-20211017T1616.tar.zst [2] https://stackoverflow.com/a/14863432/213871

[indygreg]

I agree that only the Python symbols should be exported from libpython by default. We can probably fix this by adding -fvisibility=hidden to the compilation of all packages except Python.

Since the Python extension modules are statically linked into libpython, they should bind to the OpenSSL, etc symbols appropriately.

If a foreign shared library referencing e.g. OpenSSL symbols is loaded, it won't pick up the e.g. OpenSSL symbols in libpython.

This will likely fix some issues. However, it can also cause problems since processes may have multiple versions of libraries like OpenSSL running simultaneously. This can introduce hard-to-debug failures - including run-time crashes - especially if the different library versions somehow interact with each other. I can hope this scenario will be rare. But it is bound to occur.

[ceztko]

We can probably fix this by adding -fvisibility=hidden to the compilation of all packages

Yes, that's correct. Sometimes some stupid build systems actually enforce __attribute__ ((visibility ("default"))) also when building static libraries: in these case it's possible that you still get those symbols exported by libpython even when building that dependency with -fvisibility=hidden. I hope you have no one of those. In my experience, the only solution in these cases is patching the sources so they don't enable default visibility on their symbols.

except Python

I think Python can be compiled safely with -fvisibility=hidden as well. All the public symbols should be already enabled by the machinery of the PyAPI_FUNC and the Py_EXPORTED_SYMBOL, which it does basically enable the default visibility with __attribute__ ((visibility ("default"))) similarly as it does in Windows with __declspec(dllexport).

If a foreign shared library referencing e.g. OpenSSL symbols is loaded, it won't pick up the e.g. OpenSSL symbols in libpython.

Exactly.

This will likely fix some issues. However, it can also cause problems since processes may have multiple versions of libraries like OpenSSL running simultaneously.

I think this move will fix vastly more issues than it can possibly create. Even the scenario you describe should be out of your concerns: if things go well because some code erroneously bind your symbols it's just luck.

[ceztko]

By the way, I almost got everything compiled correctly with -fvisibility=hidden put in targets.yml -> target_cflags. Pity it failed when building I think libX11 because it was attempting to build a shared object libX11.so, which I believe it should be unnecessary. I am using 3d6e597 .

[indygreg]

For many of the packages, we still build shared libraries. The mechanism by which Python statically links things is we simply delete the .so files before build/link.

I know you can tell the linker to prefer a static library over a shared one. But it was easier (and I'm lazy) to just use default build system settings in most places and rip out the shared libraries at the last minute. Having the shared libraries also keeps the door open for more traditional dynamic Python distributions. I always wanted to extend support for the "extension module variants" feature to allow providing both static and shared extension module variants as well as extension module variants not statically linking their library dependencies. So having the dynamic libraries could provide future value.

But since none of this extension module variant stuff has materialized yet, I think it is fine to just change the build scripts for libX11 (and anything else) to not produce a shared library if that makes this feature easier to implement.

[indygreg]

I also wonder if making symbols hidden might magically fix #95...

[ceztko]

I updated my initial logs to use the output of nm -D instead, since I noticed that readelf still shows local use of dynamic simbols (LOCALas opposed to GLOBAL). nm -D output is much more readable and only shows globally available symbols. No difference in the analysis conclusion: standalone python shared object builds export more symbols than necessary.

[ceztko]

As a proof of concept, I was able to produce a stripped standalone libpython3.9.so with the following ugly patch that applies to 3d6e597 :

visibility-hidden-patch-proof-of-concept.txt

This build produces the following output with nm -D: standalone-libpython3.9-nm-stripped.txt

For some reasons, some T (text) section symbols are from openssl are still present. These symbols can be further stripped by appending -Wl,--exclude-libs,ALL to LDFLAGS in build-cpython.sh, that produces the following nm -D output: standalone-libpython3.9-nm-stripped-exclude-libs.txt

As said -Wl,--exclude-libs,ALL is not available under mac.

I can confirm stripping unwanted symbols from the standalone libpython3.9.so produces a working binary and fixes the crashes I was observing that involved other software that was dynamically loading openssl symbols in the same process.

[ceztko]

Also:

• I confirm -fvisibility=hidden is safe (and recommendable) for compiling libpython too, unless one really wants to recklessly use python private symbols;
• I understood why openssl still export some symbols, even when compiled with -fvisibility=hidden: openssl defines a lot of assembly routines and manually exports them. To avoid having them in the final libpython.so there are no other ways to patch assembly sources (which is crazy), compile with linker option -Wl,--exclude-libs,ALL (which is unavailable under mac), or compile openssl without asm routines (which is less performant).

I just asked on the availability of --exclude-libs in apple ld64 in the darwin-dev[1] mailing list.

[1] https://groups.google.com/g/darwin-dev/c/GgKQnNcPTHk

[indygreg]

Thank you for your continued work on this!

Everything you say seems reasonable to me. OpenSSL assembly symbol visibility is wonky. We should ship the assembly implementations because those can have performance implications. So I guess that means we must use -Wl,--exclude-libs,ALL.

The only other thing I can think of that may be a factor here is whether libpython still exports the full, non-limited set of symbols (see PEP 384 and PEP 652). I'm pretty sure CPython has macros for defining stable vs limited APIs and these use symbol visibility builtins. We want to be sure that libpython is still exporting non-limited symbols since downstream consumers (like PyOxidizer) rely on these non-limited APIs.

[ceztko]

Thank you for your continued work on this!

You're welcome. In the mean time an answer in the darwin-dev mailing list is pointing out that -hidden-lx switch[1] has been (recently) added to the ld64 linker. Unfortunately it hasn't the same semantics as --exclude-libs but it should help achieve the same results in the apple builds.

[1] https://stackoverflow.com/a/70425525/213871

[indygreg]

I'm hacking on this a bit today. Expect to see some commits referencing this issue appear in the issue timeline soon.

After disabling a binary in the tk package that was pulling in libX11.so and refusing to compile due to new symbol visibility, I got every package - including tcl/tk - building with -fvisibility=hidden. On the Python side of things, I just had to filter out -fvisibility=hidden from CFLAGS (CPython's build system does the right thing as far as I can tell) and add -Wl,--exclude-libs,ALL to LDFLAGS to remove some symbols from OpenSSL, liblzma, TCL, and a few other libraries from the non-hidden set.

I've only implemented this on Linux so far. But I'm optimistic we can get it working for Apple targets easily enough.

[indygreg]

Getting this to work on macOS is going to entail a bit more effort. I think it is doable by using ld64 + -hidden-l. But it entails switching (I believe) to the system linker instead of our provided modern Clang linker. I'd strongly prefer to use our custom toolchain for a few reasons. One is that we build with an older macOS SDK in CI because the SDK we build with determines which minimum SDK can be used to re-link the distributed object files. That means we would be forced to use an older linker for macOS.

It's really too bad open source Clang/lld doesn't seem to have an easy-to-use option for influencing visibility on macOS. (Although llvm-jitlink does have a -hidden-l argument, interestingly.)

All that being said, I don't believe symbol visibility matters on modern Apple platforms that much. I'm far from an expert here so please call me out if I'm wrong, but I believe modern Mach-O always uses two-level namespaces and two-level namespaces require that every undefined symbol be associated with a lookup hint. Practically speaking, every undefined symbol is annotated with the library that provides it. More info at https://github.com/aidansteele/osx-abi-macho-file-format-reference#twolevel_hints_command.

So, if you have two Mach-O libraries loaded into the same process, they can both define overlapping symbols but due to the library annotations the loader will resolve symbols to the appropriate library. I believe the only scenario where the presence of unwanted global symbols in our distributions can cause problems is when our libpython (or its object files) are linked (not loaded) against other object files / libraries. In this scenario, our symbols may get used. There are pros and cons to this.

Anyway, I think I talked myself into not doing anything with the Apple distributions for the moment. I remain receptive to changing my mind on this if someone can convince me the visible symbols on macOS are actually a problem.

[ceztko]

But it entails switching (I believe) to the system linker instead of our provided modern Clang linker. I'd strongly prefer to use our custom toolchain for a few reasons.

I don't know exactly what linker you're using for mac, I would be curious to know. I think lldb, which should be the official llvm suite linker, supports --exclude-libs[1], but it has still incomplete Mach-o support.

[1] https://reviews.llvm.org/D34422

[indygreg]

We use the clang driver from our custom Clang toolchain as the driver for the linker. And adding -v to LDFLAGS says that this is actually using the system linker (as opposed to lld from our custom Clang toolchain). The actual link argument for libpython.dylib is something like this:

 "/usr/bin/ld" -demangle -lto_library /private/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmpp0da8hu9/tools/clang-macos/lib/libLTO.dylib -dynamic -dylib -arch arm64 -platform_version macos 11.0.0 12.3 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -undefined dynamic_lookup -undefined dynamic_lookup -o libpython3.9.dylib -L/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmpp0da8hu9/tools/deps/lib -L/install/lib -single_module -install_name /install/lib/libpython3.9.dylib -compatibility_version 3.9 -current_version 3.9 Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/token.o Parser/pegen/pegen.o Parser/pegen/parse.o Parser/pegen/parse_string.o Parser/pegen/peg_api.o Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/peephole.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/_abc.o Modules/_asynciomodule.o Modules/_bisectmodule.o Modules/_bz2module.o Modules/_codecs_cn.o Modules/_codecs_hk.o Modules/_codecs_iso2022.o Modules/_codecs_jp.o Modules/_codecs_kr.o Modules/_codecs_tw.o Modules/_codecsmodule.o Modules/_collectionsmodule.o Modules/_contextvarsmodule.o Modules/_cryptmodule.o Modules/_csv.o Modules/_ctypes.o Modules/_ctypes_test.o Modules/_curses_panel.o Modules/_cursesmodule.o Modules/_datetimemodule.o Modules/_dbmmodule.o Modules/_decimal.o Modules/_elementtree.o Modules/_functoolsmodule.o Modules/_hashopenssl.o Modules/_heapqmodule.o Modules/_iomodule.o Modules/_json.o Modules/_localemodule.o Modules/_lsprof.o Modules/_lzmamodule.o Modules/_math.o Modules/_opcode.o Modules/_operator.o Modules/_peg_parser.o Modules/_pickle.o Modules/_posixsubprocess.o Modules/_queuemodule.o Modules/_randommodule.o Modules/_scproxy.o Modules/_sre.o Modules/_ssl.o Modules/_stat.o Modules/_statisticsmodule.o Modules/_struct.o Modules/_testbuffer.o Modules/_testimportmultiple.o Modules/_testinternalcapi.o Modules/_testmultiphase.o Modules/_threadmodule.o Modules/_tkinter.o Modules/_tracemalloc.o Modules/_uuidmodule.o Modules/_weakref.o Modules/_xxsubinterpretersmodule.o Modules/_xxtestfuzz.o Modules/_zoneinfo.o Modules/arraymodule.o Modules/atexitmodule.o Modules/audioop.o Modules/basearith.o Modules/binascii.o Modules/blake2b_impl.o Modules/blake2module.o Modules/blake2s_impl.o Modules/bufferedio.o Modules/bytesio.o Modules/cache.o Modules/callbacks.o Modules/callproc.o Modules/cfield.o Modules/cmathmodule.o Modules/connection.o Modules/constants.o Modules/context.o Modules/convolute.o Modules/crt.o Modules/cursor.o Modules/difradix2.o Modules/dlfcn_simple.o Modules/errnomodule.o Modules/faulthandler.o Modules/fcntlmodule.o Modules/fileio.o Modules/fnt.o Modules/fourstep.o Modules/fuzzer.o Modules/grpmodule.o Modules/io.o Modules/iobase.o Modules/itertoolsmodule.o Modules/malloc_closure.o Modules/mathmodule.o Modules/md5module.o Modules/microprotocols.o Modules/mmapmodule.o Modules/module.o Modules/mpalloc.o Modules/mpdecimal.o Modules/multibytecodec.o Modules/multiprocessing.o Modules/numbertheory.o Modules/parsermodule.o Modules/posixmodule.o Modules/posixshmem.o Modules/prepare_protocol.o Modules/pwdmodule.o Modules/pyexpat.o Modules/readline.o Modules/resource.o Modules/rotatingtree.o Modules/row.o Modules/selectmodule.o Modules/semaphore.o Modules/sha1module.o Modules/sha256module.o Modules/sha3module.o Modules/sha512module.o Modules/signalmodule.o Modules/sixstep.o Modules/socketmodule.o Modules/statement.o Modules/stgdict.o Modules/stringio.o Modules/symtablemodule.o Modules/syslogmodule.o Modules/termios.o Modules/textio.o Modules/timemodule.o Modules/tkappinit.o Modules/transpose.o Modules/unicodedata.o Modules/util.o Modules/xmlparse.o Modules/xmlrole.o Modules/xmltok.o Modules/xxsubtype.o Modules/zlibmodule.o Python/frozen.o -lz -lbz2 -lncurses -lpanel -lncurses -lffi -ldl -lm -lssl -lcrypto -llzma -framework SystemConfiguration -framework CoreFoundation -lsqlite3 -lssl -lcrypto -framework AppKit -framework ApplicationServices -framework Carbon -framework CoreFoundation -framework CoreServices -framework CoreGraphics -framework IOKit -framework QuartzCore -ltcl8.6 -ltk8.6 -luuid -ledit -lncurses -ldl -framework CoreFoundation -lSystem /private/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmpp0da8hu9/tools/clang-macos/lib/clang/14.0.3/lib/darwin/libclang_rt.osx.a

So I guess we could adopt -hidden-l after all without worrying about having to swap linkers...

[indygreg]

OK. I think I got Apple/Mach-O symbols hidden now as well. Commit is up on CI. If that passes, I'll likely push to main. I added the same validation logic for Mach-O as I implemented for ELF. So I'm reasonably certain that behavior is consistent between the platforms.

Thank you for all your help on this! You definitely saved me a bit of work!

[indygreg]

And pushed commits to main to no longer export symbols. W00t.

[ceztko]

Thank you for all your help on this! You definitely saved me a bit of work!

You're welcome and thank you so much for the great work with this project which was exactly what I was looking for in a solution that uses python embedding!