--libs is inconsistent for python-config --libs and pkgconfig python --libs

[doko42]

BPO	15590
Nosy	@doko42, @vstinner, @ned-deily, @haubi, @tdsmith
PRs	python/cpython#737
Files	32143cda4d80.diff: proposed fix, although also containing diff for issue#21660

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-bug', 'build'] title = '--libs is inconsistent for python-config --libs and pkgconfig python --libs' updated_at = user = 'https://github.com/doko42' ``` bugs.python.org fields: ```python activity = actor = 'ned.deily' assignee = 'none' closed = False closed_date = None closer = None components = ['Build'] creation = creator = 'doko' dependencies = [] files = ['35485'] hgrepos = ['250'] issue_num = 15590 keywords = ['patch', 'needs review'] message_count = 11.0 messages = ['167683', '167686', '219665', '219679', '219680', '219764', '219774', '219821', '219823', '219825', '235875'] nosy_count = 6.0 nosy_names = ['doko', 'vstinner', 'ned.deily', 'Arfrever', 'haubi', 'tdsmith'] pr_nums = ['737'] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue15590' versions = ['Python 2.7', 'Python 3.4', 'Python 3.5'] ```

[doko42]

seen with at least 2.7, 3.2 and 3.3:

$ python-config --libs
-lpthread -ldl -lutil -lm -lpython2.7

$ pkg-config python --static --libs
-lpthread -ldl -lutil -lpython2.7

python-config uses the SYSLIBS macro, while pkg-config uses the LIBS macro.

depending on what you want to do, both can be wrong or right:

• to build an extension, LIBS should be used.

• to link an embedded interpreter, SYSLIBS should be used, which might not be complete. Howver in this case MODLIBS is missing.

a patch should be easy once we can define the intended behaviour.

[80036ac5-bb84-4d39-8416-02cd8e51707d]

Behavior of 'pkg-config python' is more useful since without --static option only the set of directly needed libraries is returned.

In Python 3.3 or 3.4 --static option could be introduced in python-config and behavior without this option could be changed to match 'pkg-config python'.

[bbb0ac20-71b9-4570-8406-87bff8ea1119]

Well, they should not be identical, as they are for different use cases.

"pkg-config python" is to build an application containing a python interpreter (like python$EXE): + Link against libpython.so. Additionally, + re-export symbols from libpython.so for the python-modules (platform-specific). = This is similar to build against any other library, thus using 'pkg-config python'.

"python-config" is to build a python-module (like build/lib.\<platform>-\<pyver>/*.so): + No need to link against libpython.so, instead + expect symbols from libpython.so to be available at runtime, platform specific either + as a list of symbols to import from "the main executable" (AIX), or + as undefined symbols at build-time (Linux, others), or = This is specific to python-modules, thus using 'python-config'.

[80036ac5-bb84-4d39-8416-02cd8e51707d]

They are not for different cases. Difference between is a result of the fact that they were implemented independently at different times.

Linking of extension modules against libpythonX.Y.so is a good idea anyway. It avoids build failure with LDFLAGS="-Wl,--no-undefined".

[doko42]

they are. assume you build the zlib and elementtree extensions as builtins, then libs for an interpreter includes libz and libexpat, while they are not needed for extensions.

[bbb0ac20-71b9-4570-8406-87bff8ea1119]

So this diff - depending on issue#21660 - now drops showing any $LIBS from python-config, as python-modules usually do not link against any python-known libraries.

Instead, now there is a new configure variable LINKFORMODULE, which is shown by python-config --ldflags.

And $LINKFORSHARED is added to python.pc (for pkg-config).

Eventually, this allows to drop those Darwin hackery wrt. python-framework, as the python library isn't linked against the modules any more - but to PYTHONFRAMEWORK via LINKFORMODULE instead.

However, I don't have any Darwin around here: anyone?

For AIX, this looks good so far.

Thanks!

[80036ac5-bb84-4d39-8416-02cd8e51707d]

Michael Haubenwallner: Could you show output of 'python-config-3.5 --ldflags' and 'python-config-3.5 --libs' without and with this patch (on GNU/Linux and AIX)?

[bbb0ac20-71b9-4570-8406-87bff8ea1119]

For AIX, with both these configure variants: $ configure --prefix=/prefix --enable-shared CC=gcc CXX=g++ OPT= $ configure --prefix=/prefix --enable-shared --without-computed-gotos CC=xlc_r CXX=xlC_r OPT= the output changes like this:

   $ PKG_CONFIG_PATH=/prefix/lib/pkgconfig pkg-config --libs python-3.4
old: -L/prefix/lib -lpython3.4m
new: -L/prefix/lib -lpython3.4m -Wl,-bE:/prefix/lib/python3.4/config-3.4m/python.exp -lld

   $ /prefix/bin/python3.4-config --ldflags
old:  -L/prefix/lib -lintl -ldl -lm  -lpython3.4m -Wl,-bE:Modules/python.exp -lld
new: -Wl,-bI:/prefix/lib/python3.4/config-3.4m/python.exp

For Linux, with this configure variant: $ configure --prefix=/prefix --enable-shared CC=gcc CXX=g++ the output changes like this:

  $ PKG_CONFIG_PATH=/prefix/lib/pkgconfig pkg-config --libs python-3.4
old: -L/prefix/lib -lpython3.4m
new: -L/prefix/lib -lpython3.4m -Xlinker -export-dynamic

   $ /prefix/bin/python3.4-config --ldflags
old:   -L/prefix/lib -lpthread -ldl  -lutil -lm  -lpython3.4m -Xlinker -export-dynamic
new: # Yes, nothing. Not sure if python modules should have undefined python symbols, or link against libpython.so

Just found out that distutils.command.build_ext.get_libraries() does add libpython.so for Linux and similar (not Darwin and AIX). Actually, distutils.command.build_ext.get_libraries() should add $LINKFORMODULE instead now, so LDSHARED does not need to carry.

More thoughts? (will post results with --disable-shared as separate comment)

[bbb0ac20-71b9-4570-8406-87bff8ea1119]

Now for --disable-shared:

For AIX, with both these configure variants: $ configure --prefix=/prefix --disable-shared CC=gcc CXX=g++ OPT= $ configure --prefix=/prefix --disable-shared --without-computed-gotos CC=xlc_r CXX=xlC_r OPT= the output changes like this:

   $ PKG_CONFIG_PATH=/prefix/lib/pkgconfig pkg-config --libs python-3.4
old: -L/prefix/lib -lpython3.4m
new: -L/prefix/lib -lpython3.4m -Wl,-bE:/prefix/lib/python3.4/config-3.4m/python.exp -lld

   $ /prefix/bin/python3.4-config --ldflags
old: -L/prefix/lib/python3.4/config-3.4m -L/prefix/lib -lintl -ldl -lm  -lpython3.4m -Wl,-bE:Modules/python.exp -lld
new: -Wl,-bI:/prefix/lib/python3.4/config-3.4m/python.exp

For Linux, with this configure variant: $ configure --prefix=/prefix --enable-shared CC=gcc CXX=g++ the output changes like this:

  $ PKG_CONFIG_PATH=/prefix/lib/pkgconfig pkg-config --libs python-3.4
old: -L/prefix/lib -lpython3.4m
new: -L/prefix/lib -lpython3.4m -Xlinker -export-dynamic

   $ /prefix/bin/python3.4-config --ldflags
old: -L/prefix/lib/python3.4/config-3.4m -L/prefix/lib -lpthread -ldl  -lutil -lm  -lpython3.4m -Xlinker -export-dynamic
new: # Yes, nothing. 

And with --disable-shared, even distutils.command.build_ext.get_libraries() does *not* add libpython.so for Linux and similar, so python-shipped modules are linked with python symbols undefined now - but still as shared libraries.

[bbb0ac20-71b9-4570-8406-87bff8ea1119]

Erm, the latter should read: For Linux, with this configure variant: $ configure --prefix=/prefix --disable-shared CC=gcc CXX=g++

Now reading GNU ld manpage for Linux:

$ PKG_CONFIG_PATH=/prefix/lib/pkgconfig pkg-config --libs python-3.4 new: -L/prefix/lib -lpython3.4m -Xlinker -export-dynamic

ld(1), at '--export-dynamic', tells about '--dynamic-list=file' to export specific symbols only - which actually looks similar to '-bE:file' for AIX...

[03084038-fe35-4ee8-80ec-1f76217d68cf]

On Darwin, it would be nice if LINKFORMODULE used "-undefined dynamic_lookup" instead of explicitly linking to a framework binary. Modules with explicit links to a framework cause segfaults when they are imported from a different, but compatible, framework interpreter -- i.e., building against the system Python 2.7 but importing from a user-installed Python 2.7 causes an immediate crash.

I'm a maintainer of Homebrew, a package manager for OS X, and I spend a lot of time tracking down and eliminating errant explicit framework links because they make packaging binaries much harder for us and cause crashes for users.