python / cpython

The Python programming language
https://www.python.org
Other
63.5k stars 30.41k forks source link

python-config --ldflags must not contain LINKFORSHARED ("-Xlinker -export-dynamic" on Linux) #80689

Closed vstinner closed 5 years ago

vstinner commented 5 years ago
BPO 36508
Nosy @doko42, @pitrou, @vstinner, @ned-deily
PRs
  • python/cpython#12661
  • python/cpython#12748
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields: ```python assignee = None closed_at = created_at = labels = ['3.8', 'build', '3.7'] title = 'python-config --ldflags must not contain LINKFORSHARED ("-Xlinker -export-dynamic" on Linux)' updated_at = user = 'https://github.com/vstinner' ``` bugs.python.org fields: ```python activity = actor = 'vstinner' assignee = 'none' closed = True closed_date = closer = 'vstinner' components = ['Build'] creation = creator = 'vstinner' dependencies = [] files = [] hgrepos = [] issue_num = 36508 keywords = ['patch'] message_count = 6.0 messages = ['339341', '339344', '339345', '339771', '340339', '340340'] nosy_count = 4.0 nosy_names = ['doko', 'pitrou', 'vstinner', 'ned.deily'] pr_nums = ['12661', '12748'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = None url = 'https://bugs.python.org/issue36508' versions = ['Python 3.7', 'Python 3.8'] ```

    vstinner commented 5 years ago

    python-config --ldflags must not contain LINKFORSHARED.

    Attached PR modifies python-config --ldflags to no longer include LINKFORSHARED.

    This similar change was already made on macOS: see bpo-14197.

    --

    Python build system uses a LINKFORSHARED variable, extract of configure.ac:

    # LINKFORSHARED are the flags passed to the $(CC) command that links # the python executable -- this is only needed for a few systems

    This variable is set to "-Xlinker -export-dynamic" on Linux. Extract of ld manual page for --export-dynamic option: --- When creating a dynamically linked executable, using the -E option or the --export-dynamic option causes the linker to add all symbols to the dynamic symbol table. The dynamic symbol table is the set of symbols which are visible from dynamic objects at run time.

    If you do not use either of these options (or use the --no-export-dynamic option to restore the default behavior), the dynamic symbol table will normally contain only those symbols which are referenced by some dynamic object mentioned in the link.

    If you use "dlopen" to load a dynamic object which needs to refer back to the symbols defined by the program, rather than some other dynamic object, then you will probably need to use this option when linking the program itself.

    You can also use the dynamic list to control what symbols should be added to the dynamic symbol table if the output format supports it. See the description of --dynamic-list.

    Note that this option is specific to ELF targeted ports. PE targets support a similar function to export all symbols from a DLL or EXE; see the description of --export-all-symbols below. ---

    Both configure.ac and ld manual page mention an "executable", whereas LINKFORSHARED is currently exported in python-config --ldflags. Example on Fedora 29:

    $ python3-config --ldflags
     -L/usr/lib64 -lpython3.7m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic

    The "-export-dynamic" flag causes non-obvious dynamic linking bug like the following bug in Samba which embeds Python:

    --

    History of the LINKFORSHARED variable.

    (*) Python build system uses a LINKFORSHARED variable since this commit:

    commit 7cc5abd4548629cc41d3951576f41ff2ddd7b5f7 Author: Guido van Rossum \guido@python.org\ Date: Mon Sep 12 10:42:20 1994 +0000

    Support shared library creation.

    The value of the variable changed on Linux with:

    commit b65a48e2b631f9a171e6eab699974bd2074f40d7 (HEAD) Author: Guido van Rossum \guido@python.org\ Date: Wed Jun 14 18:21:23 1995 +0000

    linux elf shlib; sys/wait.h; don't add -posix for NeXT

    Extract of the configure.in change:

    if test -z "$LINKFORSHARED" then case $ac_sys_system in hp|HP) LINKFORSHARED="-Wl,-E";; + Linux*) LINKFORSHARED="-rdynamic";; esac fi

    The variable was only used to build the "python" executable. Extract of Modules/Makefile.in (at commit 7cc5abd4548629cc41d3951576f41ff2ddd7b5f7):

    ../python: config.o $(MYLIBS) Makefile $(CC) $(OPT) config.o $(LINKFORSHARED) \ $(MYLIBS) $(MODLIBS) $(LIBS) $(SYSLIBS) -o python mv python ../python

    (*) The python-config script was created as a Python script by:

    commit c90b17ec8233009e4745dd8f77401f52c5d4a8d5 Author: Martin v. Löwis \martin@v.loewis.de\ Date: Sat Apr 15 08:13:05 2006 +0000

    Patch bpo-1161914: Add python-config.

    The following commit modified Misc/python-config.in to add LINKFORSHARED to python-config --ldflags:

    commit a70f3496203cd68d88208a21d90f0ca3503aa2f6 Author: Collin Winter \collinw@gmail.com\ Date: Fri Mar 19 00:08:44 2010 +0000

    Make python-config support multiple option flags on the same command line, rather than requiring one invocation per flag.

    Extract:

    + elif opt in ('--libs', '--ldflags'): + libs = getvar('LIBS').split() + getvar('SYSLIBS').split() + libs.append('-lpython'+pyver) + # add the prefix/lib/pythonX.Y/config dir, but only if there is no + # shared library in prefix/lib/. + if opt == '--ldflags': + if not getvar('Py_ENABLE_SHARED'): + libs.insert(0, '-L' + getvar('LIBPL')) + libs.extend(getvar('LINKFORSHARED').split()) + print ' '.join(libs)

    The following commit modified Misc/python-config.in to not add LINKFORSHARED into "libs" when built on macOS (if PYTHONFRAMEWORK is defined):

    commit ecd4e9de5afab6a5d75a6fa7ebfb62804ba69264 Author: Ned Deily \nad@acm.org\ Date: Tue Jul 24 03:31:48 2012 -0700

    Issue bpo-14197: For OS X framework builds, ensure links to the shared
    library are created with the proper ABI suffix.
    diff --git a/Misc/python-config.in b/Misc/python-config.in
    index 1d4a81d850..79f0bb14c1 100644
    --- a/Misc/python-config.in
    +++ b/Misc/python-config.in
    @@ -52,7 +52,8 @@ for opt in opt_flags:
             if opt == '--ldflags':
                 if not getvar('Py_ENABLE_SHARED'):
                     libs.insert(0, '-L' + getvar('LIBPL'))
    -            libs.extend(getvar('LINKFORSHARED').split())
    +            if not getvar('PYTHONFRAMEWORK'):
    +                libs.extend(getvar('LINKFORSHARED').split())
             print(' '.join(libs))
    

    (*) A shell version of python-config has been added by:

    commit 874211978c8097b8e747c90fa3ff41aacabe340f Author: doko@python.org \doko@python.org\ Date: Sat Jan 26 11:39:31 2013 +0100

    - Issue bpo-16235: Implement python-config as a shell script.

    Extract of Misc/python-config.sh.in at this commit:

        --ldflags)
            LINKFORSHAREDUSED=
            if [ -z "$PYTHONFRAMEWORK" ] ; then
                LINKFORSHAREDUSED=$LINKFORSHARED
            fi
            LIBPLUSED=
            if [ "$PY_ENABLE_SHARED" = "0" ] ; then
                LIBPLUSED="-L$LIBPL"
            fi
            echo "$LIBPLUSED -L$libdir $LIBS $LINKFORSHAREDUSED"
        ;;
    vstinner commented 5 years ago

    I'm a little bit scared by the idea of backporting such change in Python 2.7 and 3.7 stable branches, even if I think that it's a correct bugfix. I'm scared by the number of platforms, I don't know all these linker flags and I am not able to test all combinations:

    AC_MSG_RESULT($CCSHARED)
    # LINKFORSHARED are the flags passed to the $(CC) command that links
    # the python executable -- this is only needed for a few systems
    AC_MSG_CHECKING(LINKFORSHARED)
    if test -z "$LINKFORSHARED"
    then
        case $ac_sys_system/$ac_sys_release in
        AIX*)   LINKFORSHARED='-Wl,-bE:Modules/python.exp -lld';;
        hp*|HP*)
            LINKFORSHARED="-Wl,-E -Wl,+s";;
    #       LINKFORSHARED="-Wl,-E -Wl,+s -Wl,+b\$(BINLIBDEST)/lib-dynload";;
        Linux-android*) LINKFORSHARED="-pie -Xlinker -export-dynamic";;
        Linux*|GNU*) LINKFORSHARED="-Xlinker -export-dynamic";;
        # -u libsys_s pulls in all symbols in libsys
        Darwin/*)
            LINKFORSHARED="$extra_undefs -framework CoreFoundation"
        \# Issue bpo-18075: the default maximum stack size (8MBytes) is too
        \# small for the default recursion limit. Increase the stack size
        \# to ensure that tests don't crash
        LINKFORSHARED="-Wl,-stack_size,1000000 $LINKFORSHARED"
    
        if test "$enable_framework"
        then
            LINKFORSHARED="$LINKFORSHARED "'$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
        fi
        LINKFORSHARED="$LINKFORSHARED";;
    OpenUNIX*|UnixWare*) LINKFORSHARED="-Wl,-Bexport";;
    SCO_SV*) LINKFORSHARED="-Wl,-Bexport";;
    ReliantUNIX*) LINKFORSHARED="-W1 -Blargedynsym";;
    FreeBSD*|NetBSD*|OpenBSD*|DragonFly*)
        if [[ "`$CC -dM -E - </dev/null | grep __ELF__`" != "" ]]
        then
            LINKFORSHARED="-Wl,--export-dynamic"
        fi;;
    SunOS/5*) case $CC in
          *gcc*)
            if $CC -Xlinker --help 2>&1 | grep export-dynamic >/dev/null
            then
            LINKFORSHARED="-Xlinker --export-dynamic"
            fi;;
          esac;;
    CYGWIN*)
        if test $enable_shared = "no"
        then
            LINKFORSHARED='-Wl,--out-implib=$(LDLIBRARY)'
        fi;;
    QNX*)
        # -Wl,-E causes the symbols to be added to the dynamic
        # symbol table so that they can be found when a module
        # is loaded.  -N 2048K causes the stack size to be set
        # to 2048 kilobytes so that the stack doesn't overflow
        # when running test_compile.py.
        LINKFORSHARED='-Wl,-E -N 2048K';;
    VxWorks*)
        LINKFORSHARED='--export-dynamic';;      
    esac

    fi AC_MSG_RESULT($LINKFORSHARED)

    vstinner commented 5 years ago

    See also bpo-10112: "Use -Wl,--dynamic-list=x.list, not -Xlinker -export-dynamic".

    vstinner commented 5 years ago

    New changeset e65f01f78d7bda3013fc5be485afa87ff56511d9 by Victor Stinner in branch 'master': bpo-36508: python-config don't export LINKFORSHARED (GH-12661) https://github.com/python/cpython/commit/e65f01f78d7bda3013fc5be485afa87ff56511d9

    vstinner commented 5 years ago

    New changeset cd46b09b0863c787dd54c433fae52bd8bdfaecd0 by Victor Stinner (Miss Islington (bot)) in branch '3.7': bpo-36508: python-config don't export LINKFORSHARED (GH-12661) (GH-12748) https://github.com/python/cpython/commit/cd46b09b0863c787dd54c433fae52bd8bdfaecd0

    vstinner commented 5 years ago

    The bug is fixed in 3.7 and master (future 3.8) branches. I prefer to leave 2.7 unchanged. I close the issue.

    Python 2.7 is affected as well, but I'm really scared to touch the build system of Python 2.7 which is very stable. Even for Python 3.7, I wasn't fully comfortable to merge my fix.