AIX shared library fix

[8714724b-13af-470f-a469-43d08e7c1069]

BPO	941346
Nosy	@warsaw, @birkenfeld, @pitrou, @benjaminp, @tardigradus, @haubi
Files	patch_AIX_shared.diff Python-2.6.6_shared_AIX.diff: Patch to correct compilation of Python 2.6.6 as a shared library on AIX python266_AIX61_static_gcc_nopatch.txt.gz: Build and tests log for pristine Python compiled statically on AIX 6.1 with gcc python266_AIX61_static_gcc_patchshared.txt.gz: Build and tests log for patched Python compiled statically on AIX 6.1 with gcc python266_AIX61_static_xlc_nopatch.txt.gz: Build and tests log for pristine Python compiled statically on AIX 6.1 with xlc python266_AIX61_static_xlc_patchshared.txt.gz: Build and tests log for patched Python compiled statically on AIX 6.1 with xlc python266_AIX61_shared_gcc_nopatch.txt.gz: Build and tests log for pristine Python compiled dynamically on AIX 6.1 with gcc python266_AIX61_shared_gcc_patchshared.txt.gz: Build and tests log for patched Python compiled dynamically on AIX 6.1 with gcc python266_AIX61_shared_xlc_nopatch.txt.gz: Build and tests log for pristine Python compiled dynamically on AIX 6.1 with xlc python266_AIX61_shared_xlc_patchshared.txt.gz: Build and tests log for patched Python compiled dynamically on AIX 6.1 with xlc Python-3.1.2_shared_AIX.diff: Patch to correct compilation of Python 3.1.2 as a shared library on AIX python312_AIX61_static_nopatch.txt.gz: Build and tests log for pristine Python 3.1.2 compiled statically on AIX 6.1 with xlc python312_AIX61_static_patch_shared.txt.gz: Build and tests log for patched Python 3.1.2 compiled statically on AIX 6.1 with xlc python312_AIX61_shared_patch_shared.txt.gz: Build and tests log for patched Python 3.1.2 compiled dynamically on AIX 6.1 with xlc aix32.diff Python-2.7_shared_AIX.diff: Patch to correct compilation of Python 2.7 as a shared library on AIX python312_AIX61_static_gcc_patchshared.txt.gz: Build and tests log for patched Python 3.1.2 compiled statically on AIX 6.1 with gcc python312_AIX61_shared_gcc_patchshared.txt.gz: Build and tests log for patched Python 3.1.2 compiled dynamically on AIX 6.1 with gcc py3k_aix61_static_xlc_nopatch.txt.gz py3k_aix61_static_xlc_patchshared.txt.gz py3k_aix61_static_gcc_patchshared.txt.gz py3k_aix61_shared_gcc_patchshared.txt.gz py3k_aix61_shared_xlc_pathshared.txt.gz patch_aix_shared_32.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/birkenfeld' closed_at = created_at = labels = ['extension-modules', 'type-feature', 'build'] title = 'AIX shared library fix' updated_at = user = 'https://bugs.python.org/dickdunbar' ``` bugs.python.org fields: ```python activity = actor = 'georg.brandl' assignee = 'georg.brandl' closed = True closed_date = closer = 'georg.brandl' components = ['Build', 'Extension Modules'] creation = creator = 'dickdunbar' dependencies = [] files = ['11081', '18687', '18688', '18689', '18690', '18691', '18692', '18693', '18694', '18695', '18762', '18763', '18764', '18765', '18767', '18768', '18796', '18797', '18818', '18819', '18821', '18822', '18832', '20764'] hgrepos = [] issue_num = 941346 keywords = ['patch'] message_count = 38.0 messages = ['60482', '60483', '70894', '70896', '70902', '88115', '88116', '115262', '115279', '115375', '115703', '115707', '115708', '115709', '115729', '115731', '115734', '115865', '115876', '115987', '116040', '116042', '128214', '128281', '128586', '128591', '128592', '128593', '128594', '128596', '128597', '128602', '128634', '128638', '128639', '128640', '128652', '129355'] nosy_count = 9.0 nosy_names = ['barry', 'nnorwitz', 'georg.brandl', 'pitrou', 'sable', 'benjamin.peterson', 'dickdunbar', 'loris', 'haubi'] pr_nums = [] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue941346' versions = ['Python 2.6', 'Python 3.1', 'Python 2.7'] ```

[8714724b-13af-470f-a469-43d08e7c1069]

By default, the ld command only looks for *.a archives to resolve library references.

To recognize dynamic shared libaries (*.so), two flags need to be added to the Modules/ld_so_aix script: -brtl -bnortllib

Line 170 in the script should read:

CCOPT="$CCOPT -Wl,-bM:SRE -Wl,-T512 -Wl,-H512 -brtl -bnortllib -lm -o $objfile"

-brtl ... says you want to include libtk8.4.so when searching for the library -ltk8.4

-bnortllib ... says you do not want the semantics of the AIX "run time linking" services ... just the search for the library.

This resolves a problem when Python is built with tk support.

[d21744ff-f396-4c71-955e-7dbd2e886779]

Logged In: YES user_id=33168

I added a comment based on the above information. I don't want to make a real change, since I can't test it. If we get access to some AIX boxes, we could really fix. I'm hestitant to break stuff without testing.

Checked in as:

Modules/ld_so_aix 2.6

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Hi,

I have modified a bit this patch to make it work with Python 2.5.2 (see attached patched).

It is a great enhancement for Python users on AIX to be able to use a shared library instead of a static one, so I would appreciate if this feature could be integrated in trunk.

We have various AIX 5.2 and 5.3 servers in the Sungard company where I work, and we could run some tests according to your instructions. It may also be possible to provide some limited access to one of those servers if needed but it would require some time and paper work.

What kind of access would you need in order to validate those patchs?

thanks in advance

[pitrou]

Sébastien, what does your patch change exactly? Does it build a shared lib for Python by default? If that's the case, I'm not sure it's a good idea due to AIX's particular way of treating shared libraries; it might break e.g. when trying to embed Python in another program. (I say this while being an AIX novice, but I did have to compile Python under AIX for embedded use in another third-party software... unfortunately right now I don't have access to an AIX machine with a compiler)

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Hum, I tried to recompile Python without the --enable-shared option on AIX and it crashed; so this patch may not be compatible with a static version of Python (though I have other patches in my version - cf issue 3526 - which may be incompatible).

I will do a bit of cleanup in my environment and try to make a version of the patch that works also when compiling a static Python.

[8ae5edae-60af-47b7-b563-f95efc294fe2]

Shouldn't line 170 be

CCOPT="$CCOPT -Wl,-bM:SRE -Wl,-T512 -Wl,-H512 -Wl,-brtl -Wl,-bnortllib -lm -o $objfile"

instead of

CCOPT="$CCOPT -Wl,-bM:SRE -Wl,-T512 -Wl,-H512 -brtl -bnortllib -lm -o $objfile"

Otherwise the -brtl and -bnortllib flags are interpreted as compiler flags and errors like the following will occur:

gcc: unrecognized option '-brtl'

[8ae5edae-60af-47b7-b563-f95efc294fe2]

Addition to last message:

More importantly, without "-Wl", the flags are not passed to the linker and the tcl/tk libs will not be found.

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Hi,

As reported in this issue and bpo-1756343 and bpo-1542544, Python does not produce a shared python library on AIX even with the --enable-shared flag.

I had provided a patch to correct that, but it was breaking static compilation of Python on AIX.

We are upgrading our AIX build environment to AIX 6.1 and Python 2.6.6, so I took the time to review this patch again and to clean it so that it works well when compiling Python statically and dynamically, with either gcc and xlc_r.

I attach the resulting patch, and the build and unit test logs. You can notice that not only this patch allows to compile Python dynamically on AIX, but also that it improves the unit tests results when compiling Python statically.

For example with the xlc_r compiler and Python compiled statically, the results without the patch are: 274 tests OK. 22 tests failed 70 tests skipped The results with the patch are: 286 tests OK. 21 tests failed 59 tests skipped

A minor issue with this patch is that "libpython2.6.so" is hardcoded in the ld_so_aix file. That should be modified by moving ld_so_aix to ld_so_aix.in and using $LDLIBRARY, or by using an explicit flag when calling ld_so_aix to compile libpythonx.y.so.

If you agree to the principle of this patch, I can make the extra work to clean ld_so_aix and test this patch on Python 2.7 and 3.1.

regards

[pitrou]

In dynamic builds, there seem to be lots of messages such as:

ld: 0711-224 WARNING: Duplicate symbol: .PyLong_FromString ld: 0711-224 WARNING: Duplicate symbol: PyLong_FromString

Are you sure this is normal?

In any case, if you find a way to cleanup ld_so_aix, this should probably be checked in anyway. I doubt anybody else tests under AIX, so it's reasonable to trust you on this.

Even better if you manage to diagnose and/or fix some of the test failures, by the way. (if not, you can still open separate issues for each of them)

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Le 01/09/2010 00:16, Antoine Pitrou a écrit :

In dynamic builds, there seem to be lots of messages such as:

ld: 0711-224 WARNING: Duplicate symbol: .PyLong_FromString ld: 0711-224 WARNING: Duplicate symbol: PyLong_FromString

Are you sure this is normal?

This is not normal, but I always had this warning and this does not prevent the libraries from working OK. I will try to see if I can clean that.

In any case, if you find a way to cleanup ld_so_aix, this should probably be checked in anyway. I doubt anybody else tests under AIX, so it's reasonable to trust you on this.

Even better if you manage to diagnose and/or fix some of the test failures, by the way. (if not, you can still open separate issues for each of them)

Great! this is the kind of answer I was looking for. I will cleanup my patch to make it compatible with Python 2.7 and Python 3.1.2 and test it with those 2 versions.

I will also correct as many unitary tests as possible with this platform. I already have a patch for bpo-4026.

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Hi,

I have updated the patch and adapted it for Python 3.1.2:

• this time ld_so_aix is moved to ld_so_aix.in which is converted to ld_so_aix by 'configure' so that it has the right version number for the python library

• there are no more warnings for 'Duplicate symbol': I changed build_ext.py to not link extensions with libpython since the symbols are already defined by python.exp

• I made a modification to ld_soaix specifically for Python 3 as according to pep-3121 the entry function has been renamed from init\<modulname> to PyInit\<modulename>

I will provide another patch for Python 2.7 (nearly the same, but without the modification needed by pep-3121). I also attach the build log for Python 3.1.2 without and with this patch.

regards

[pitrou]

Thank you. For the record, here is the patch adapted for 3.2 development branch. It looks ok to me.

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Here is the patch for Python 2.7.

The only difference compared to Python 3.1.2 is that we don't rename init\<modulname> to PyInit_\<modulename>.

[pitrou]

• there are no more warnings for 'Duplicate symbol': I changed build_ext.py to not link extensions with libpython since the symbols are already defined by python.exp

Could this affect building extensions with gcc under AIX? Or does gcc delegate to the AIX linker?

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Yes Antoine, gcc only handles compilation; the linker is explicitly called through the ld_so_aix script which handles calling the native ld with the right flags to import symbols.

I will check with gcc and attach the log on Wednesday (not at work tomorrow) but there should be no problem.

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Hum, forget my previous note;I checked ld_so_aix and it actually calls $CC to handle linking not ld. I suppose gcc will call the native ld.

Anyway, I will run the test with gcc on Wednesday and know for sure if that works.

[pitrou]

Anyway, I will run the test with gcc on Wednesday and know for sure if that works.

Thank you! If it works, it's good for commit.

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Here are the build logs for Python 3.1.2 modified with this patch and compiled with gcc. The extensions are correctly built.

I also compiled an extension not directly provided in Python source (cx_Oracle) and it worked fine also.

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Antoine, I wanted to test this improvement (and others) on the branch py3k (I was using Python 2.7 and Python 3.1.2 for the moment). But I have some issues compiling this branch, even without any patch (see bpo-9799).

So you may want to wait for the other issue to be resolved before to commit this patch on branch py3k.

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Antoine, I tested this patch on py3k with both gcc and xlc in static and shared mode and I did not notice any issue.

I attach the build and test logs.

I think you can safely commit it.

[pitrou]

I've committed the latest patch in r84680 (3.x), r84682 (3.1) and r84683 (2.7). Perhaps you want to check it's ok. Thank you anyway!

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Great! Thanks Antoine.

I checked quickly and there is a small correction to do on the 2.7 branch: this branch is different because there has been a change between python 2.x and 3.x in pep-3121 concerning the name of the entry function in a module (init\<modulname> to PyInit_\<modulename>)

So you need to change this line in Modules/ld_so_aix.in from: entry=PyInit_echo $filename | sed "s/module.*//" to entry=initecho $filename | sed "s/module.*//"

Except for that it looks fine.

I will continue to try to make all unitary tests pass on AIX. And I would like to setup a buildbot that will compile and test the trunk every day and post the log on a public web site, in order to help keep this platform well supported.

[bbb0ac20-71b9-4570-8406-87bff8ea1119]

Sébastien, why did you need to add '-L$(srcdir)' to BLDSHARED in these patches?

As LDLIBRARY points to the immediate file 'libpython$(VERSION).so' instead of '-lpython$(VERSION)', I don't see the need for '-L$(srcdir)' at all.

The problem is that '-L$(srcdir)' adds '$(srcdir)' to the runpath too (as there is no '-blibpath' argument), opening a security hole for libpythonX.Y.so as well as the modules.so.

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Hi Micheal,

I don't remember why "-L\$(srcdir)" was added since it has been quite a long time.

I tried to recompile everything after removing it and everything compiled just fine, so I guess it can be removed.

regards

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Hum, I was incorrect in previous note: You can remove -L\$(srcdir) on this line, but then you need to specify the full path to Modules/python.exp. See patch below:

Index: configure.in \===================================================================

--- configure.in        (revision 88422)
+++ configure.in        (working copy)
@@ -1642,7 +1655,7 @@
 then
        case $ac_sys_system/$ac_sys_release in
        AIX*)
-               BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) -bI:Modules/python.exp -L\$(srcdir)"
+               BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) -bI:\$(srcdir)/Modules/python.exp"
                LDSHARED="\$(BINLIBDEST)/config/ld_so_aix \$(CC) -bI:\$(BINLIBDEST)/config/python.exp"
                ;;
        IRIX/5*) LDSHARED="ld -shared";;

Also, I think there are other issues with building Python 3.2 with shared libraries. I am currently investigating.

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

OK, Python with shared libraries is broken in trunk since the library was renamed to libpython3.2m. Here is a patch to correct that:

Index: Modules/ld_so_aix.in \===================================================================

--- Modules/ld_so_aix.in        (revision 88422)
+++ Modules/ld_so_aix.in        (working copy)
@@ -131,7 +131,7 @@
   shift
 done

-if test "$objfile" = "libpython@VERSION@.so"; then
+if test "$objfile" = "libpython@VERSION@@ABIFLAGS@.so"; then
   ldsocoremode="true" 
 fi

[birkenfeld]

This looks like it should (and could) go into 3.2 final. Agreed?

[pitrou]

Agreed.

[birkenfeld]

OK. Sébastien, could you make and attach a complete patch?

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Here is the full patch. I also modified Makefile.pre.in to clean Modules/python.exp when doing distclean.

[pitrou]

Looks good to me.

[birkenfeld]

Committed in r88426.

[bbb0ac20-71b9-4570-8406-87bff8ea1119]

You can remove -L\$(srcdir) on this line, but then you need to specify > the full path to Modules/python.exp. See patch below:

Interesting, didn't experience this to be necessary with Python-2.7.1 here... Maybe because I do an in-source build, haven't tried an out-of-source build at all.

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Are you using --enable-shared when compiling python? That is when you should see the error (it could not find python.exp at some stage).

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

By the way... thanks Georg!

[cf2aeb3c-f43f-407a-8d37-30a2b5033222]

Also Michael, I am working on the 3.2 branch, it may be different with Python 2.7.1 but I haven't tested yet.

[pitrou]

Georg, the non-abiflags portion seems to need backporting.

[birkenfeld]

Backported to 3.1 in r88560, 2.7 in r88568.