BPO	27435
Nosy	@haubi, @vadmium, @zware, @aixtools
PRs	python/cpython#5156
Files	issue27435_ctypes_aix_support.patch Python2.7.Lib.ctypes.160928.patch: patch compared to Python-2.7.12 tarball _aix.py: The platform dependent processing for find_library() on AIX

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['ctypes', 'type-bug'] title = 'ctypes library loading and AIX - also for 2.7.X (and later)' updated_at = user = 'https://bugs.python.org/aixtoolsgmailcom' ``` bugs.python.org fields: ```python activity = actor = 'zach.ware' assignee = 'none' closed = True closed_date = closer = 'zach.ware' components = ['ctypes'] creation = creator = 'aixtools@gmail.com' dependencies = [] files = ['43935', '44868', '44869'] hgrepos = [] issue_num = 27435 keywords = ['patch'] message_count = 23.0 messages = ['269673', '271513', '271631', '271741', '271820', '271821', '272453', '273307', '273326', '273392', '273414', '273458', '277633', '282233', '284557', '286548', '286889', '288202', '288509', '295900', '309430', '323599', '367385'] nosy_count = 5.0 nosy_names = ['haubi', 'martin.panter', 'zach.ware', 'David.Edelsohn', 'Michael.Felt'] pr_nums = ['5156'] priority = 'normal' resolution = 'out of date' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue27435' versions = ['Python 2.7'] ```

I am opening a new issue # - about the same problem described in 26439, but now calling it a bug rather than a behavior change. Then I did not know better - all was new, and as the "new" kid I felt uncomfortable calling it a bug - when maybe it was just behavior I did not understand

However, with the feedback from code review I am beginning to understand what the ctypes functionality is: ctypes.find_library() and cdll.LoadLibrary (aka CDLL)

in __init__.py it is clear that there is one entry point for LoadLibrary. For find_library(), I assume overtime, the development led to six (6) definitions - and I am proposing a 7th.

The pseudocode is roughly this (although import ctypes.\$_platform._xxx may be incorrect syntax) def findlibrary(name) import os,sys _platform = sys.platform import ctypes.\$_platform._find_library as _find_library return(_find_library(name))

Unclear is how a regular .so file should be treated by find_library() when the actual file is not prefixed by the string "lib", e.g., sudoers.so - should it only look for libsudoers.so even though

Currently it is "defined" 7 times (including the definition I added at line 79) because - again - util.py defines find_library several times - only one of the definitions is called dependent on the os.name or os.name + platform.

Within the os.name() and sys.platform code block there are sufficient "liberities" to call external programs used to determine where (i.e., find) the library is - and what it's correct prefix and suffix is.

+6  # find_library(name) returns the pathname of a library, or None.

+49 def find_library(name): in block if os.name == "nt" +71 def find_library(name): if os.name == "ce" return None +79 def find_library(name): if sys.platform.startswith("aix") return ctypes._aixutils.find_library(name) +84 def find_library(name): elif os.name == "posix" and sys.platform == "darwin": +95 elif os.name == "posix": +177 def find_library(name): inside: if (sys.platform.startswith("freebsd") or sys.platform.startswith("openbsd") or sys.platform.startswith("dragonfly")):

+217 def find_library(name, is64 = False): inside: elif sys.platform == "sunos5":

+220 else: +249 def find_library(name): +250 return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))

++++++ Getting back to my patch development - initially I was adding code to both util.py and __init__.py and putting everything in " startswith("aix")" blocks. As I thought, in error, find_library() was to make visible what CDLL was finding on it's own I was also doing find_library() like behavior. Rather than have the same code in two files I moved the code into a new file - now named _aixutils.py.

So, this looks like a design change, but it is not. All this code could be re-indented and inserted into util.py. IMHO, in a separate file the changes are easier to follow. If a seperate file means it cannot be accepted into python-2.7.X then the code needs to be moved back into util.py

Note: the extra tests added to util.py are only for my enjoyment, i.e., verification of several cases I have experienced while porting other packages. Think of them as not being there (for now, if not for always).

Once I knew what CDLL(None) was suppossed to That is why my first patch had CDLL() also calling ctypes._aixutils - to only edit once. Once I learned how wrong I was - that was removed - and it just remained this way as "eye-candy".

+++ Closing +++

find_library() is defined multiple times, so adding one for aix is not a design change, i.e., behavior modification rather, find_library - as defined for aix in "default block" (elif os.name = "posix": ... else: ...) specifically calling _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)) is a "bug of ommission" because the programs the two routines called depend on: ldconfig and gcc are not, normally present on AIX. My patch is a different way changing line 250 from return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)) to return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)) or _findAIXmember(name) or _find_Soname_dump(name)

with, of course def _findAIXmember() and def _findSoname_dump() defined within the limits of util.py Instead, I have "hard-coded" an approach similar to the psuedo code example.

+++ Disclaimer :P +++ As I am still relatively new, if I am still not understanding the official definition of ctypes.find_library() - I apologize. util.py is, due to is many defines of find_library() was difficult for me to grasp the difference between permissible and "not done".

Going back to the earliest documentation I could find for ctypes https://docs.python.org/2.6/library/ctypes.html (new in 2.5) the key lines are:

ctypes exports the cdll, and on Windows windll and oledll objects, for loading dynamic link libraries.

You load libraries by accessing them as attributes of these objects. cdll loads libraries which export functions using the standard cdecl calling convention, while ...

For find_library() - the documentation is quite clear - find_library is system dependent - mentioning that "linux", as one example, uses ldconfig to find libraries:

15.15.2.1. Finding shared libraries

When programming in a compiled language, shared libraries are accessed when compiling/linking a program, and when the program is run.

The purpose of the find_library() function is to locate a library in a way similar to what the compiler does (on platforms with several versions of a shared library the most recent should be loaded), while the ctypes library loaders act like when a program is run, and call the runtime loader directly.

The ctypes.util module provides a function which can help to determine the library to load.

ctypes.util.find_library(name)
    Try to find a library and return a pathname. name is the library name without any prefix like lib, suffix like .so, .dylib or version number (this is the form used for the posix linker option -l). If no library can be found, returns None.

The exact functionality is system dependent.

On Linux, find_library() tries to run external programs (/sbin/ldconfig, gcc, and objdump) to find the library file. It returns the filename of the library file. ....

In short, I believe there is sufficient clarity in the initial documentation - that it is an "error of ommission" (so maybe "shame on IBM" that noone took this up years ago) and has every right as Linux, Solaris, openBSD, etc.. to have the system dependent glue that is needed to work "natively" on the platform. -- but I am biased -- so if I am wrong - please be kind.

Note: goal is to have the new patch, still with _aixutil.py as a matter of textual clarity rather than as language/module syntax. If being in a different file is the primary/only objection then, when I am again able to devote time - I will move the code into an "aix" block within util.py

Adding "type behavior" as I have now read that that is actually the python friendly way of talking about a 'bug'.

Testing my proposed patch for 2.7 with python2.7.12 - will update with patch when finished.

""" Lib/ctype support for LoadLibrary interface to dlopen() for AIX Similar kind of support (i.e., as a separate file) as has been done for Darwin support ctype.macholib.* rather than as been done with for "ce", "nt", "solaris", and "bsd" with separate, detailed if: sections in utils.py Author: M Felt, aixtools.net, July 2016 """

FYI: I tried to pip install Mercurial - but got an error message from an include file so the last bit did not compile. Problem for a later date - could be a user error on my part, or an AIX version dependency. (during discussion bpo-26439)

I have mercurial "installed", but not (yet) any idea on how to use it to submit a patch.

Again: going back to comments in bpo-26439 - and my 'new' responses as far as the patch submitted (but not yet using Mercurial)

Here is a summary of what I would be comfortable adding to Python:

Add ctypes.RTLD_MEMBER constant (new API, so 3.6+ only)

Not included. RTLDMEMBER is a well documented constant needed for AIX dlopen. It is hard-coded in \_init__.py

Basic ctypes.util.find_library("crypto") implementation for AIX. But cases like find_library("libintl.so") should fail. I think your earlier patches were closer to this implementation than the later ones.

Months further, I understand that is a variant behavior, and has been removed. find_library("c"), find_library("crypto") are supported, an argument suchas find_library("libintl.so") will return None (or something like /usr/lib/liblibintl.so will need to exist).

I am a bit hesitant about the automatic behaviour of CDLL("libcrypto.a(libcrypto.so.1.0.0)") using RTLD_MEMBER. IMO it may be better to let the caller specify RTLD_MEMBER explicitly. If a shared library file literally called “/usr/lib/libcrypto.a(libcrypto.so.1.0.0)” existed, i.e. not inside an archive, would dlopen("libcrypto.a(libcrypto.so.1.0.0)", RTLD_NOW) succeed? I admit this is an unlikely scenario, but it seems bad to reduce the domain of a low-level API.

I hope this is now acceptable (see http://bugs.python.org/issue26439#msg267254) > ## Automatic RTLD_MEMBER ##

I was still uneasy about the automatic setting of RTLD_MEMBER. But I looked for how others handle this, and I found Libtool’s LTDL library, and Apache Portable Runtime (APR). Both have a similar (but stricter) automatic addition based on detecting the archive(member) notation:

http://git.savannah.gnu.org/cgit/libtool.git/commit/libltdl/loaders/dlopen.c?id=8fa719e https://github.com/apache/apr/commit/c27f8d2

So I guess I can accept this way,

I understand it would be good to have the return value of find_library() consistent with the name accepted by CDLL(). Perhaps a new parameter format would help, such as a tuple (archive, member).

I am not comfortable with other aspects. I think I would have to see more discussion with other people to change my opinion:

CDLL("libintl.so") should not load “libintl.a(libintl.so.1)”. I understand you want this to help code written for Gnu or Linux automatically work on AIX, but it doesn’t feel correct and robust to me. Perhaps moving this sort of thing to a separate function or package would be better.

find_library("libintl.so") -> "libintl.a(libintl.so.1)". I would expect it to look for a shared library installed in something like "/usr/lib/liblibintl.so.a", unless I have misunderstood how compile-time linking (cc -llibintl.so) works.

Again, my misunderstanding of how this function is intended - python newbie. arguments starting with "lib" are not likely to work - and it is not on find_library to strip "lib" to make the argument more -l like.

find_library() should not set the LIBPATH environment variable.

As before, and as is the status in bpo-9998 - no use of LIBPATH or LD_LIBRARY_PATH - even though dlopen() does use them. Although, if your objection is primarily on 'set' (i.e., a get could be accepted) - that is easy to put back in.

Thanks again for your help.

Note: the reason for the new issue# is because too much of what I was submitting before was "extension" and not "correction".

Hopefully, this makes the discussion more clear.

For 2.7, adding the automatic RTLD_MEMBER mode does not seem like a bug fix to me. Currently, I understand this code could load two separate libraries:

file = CDLL("libcrypto.a(libcrypto.so.1.0.0)")
member = CDLL("libcrypto.a(libcrypto.so.1.0.0)", DEFAULT_MODE | 0x00040000)

With your proposed change, the first line will do the same as the second line, and Python will no longer provide a way to load a file named like in the first operation. Maybe this is okay for the next version of Python 3 (because it is only breaking a rare corner case), but my view is it is not worth changing 2.7.

If you are still figuring out Mercurial, maybe see \https://docs.python.org/devguide/setup.html#getting-the-source-code\ and \https://docs.python.org/devguide/patch.html#tool-usage\ if you haven’t already.

On 02-Aug-16 15:34, Martin Panter wrote:

Martin Panter added the comment:

For 2.7, adding the automatic RTLD_MEMBER mode does not seem like a bug fix to me. Currently, I understand this code could load two separate libraries:

file = CDLL("libcrypto.a(libcrypto.so.1.0.0)") Noone (in their right mind, imho) would install, or extract the archive member libcrypto.so.1.0.0and then rename it /usr/lib/libcrypto.a(libcrypto.so.1.0.0)

FYI - I did test this case, and as a file, without 0x000400000 (aka RLTD_MEMBER) or'd into the mode.

member = CDLL("libcrypto.a(libcrypto.so.1.0.0)", DEFAULT_MODE | 0x00040000)

With your proposed change, the first line will do the same as the second line, and Python will no longer provide a way to load a file named like in the first operation. Maybe this is okay for the next version of Python 3 (because it is only breaking a rare corner case), but my view is it is not worth changing 2.7. Right now it is broken in AIX - it is impossible to use the native archive(member) support. Direct loading of .so files, if I recall correctly, was to provide linux affinity (remember the L in AIX 5L).

I believe python2 has some years to go, and basically, you ask anyone using python on AIX to go through all kinds of loops. This hurts python acceptance on AIX - too much work to get it working and keep it working.

I have been providing AIX support for over 20 years - NEVER have I seen anyone name a shared library libxxx.a(libxxx.so). What I have seen is that people extract archive members from a .a archive into the same directory - but when the archive gets updated most forget to extract the members again.

In yet another case I have seen a case where they copied everything to a new directory and do a chroot() to get the .so files they want - because it is impossible to load from a .a file.

Yes, when I first started back in February and March: a) knew next to nothing about python; b) was trying to solve it in ways I would like it be (mainly more flexible aka smarter re: the argument to find_library().

However, I do believe what I have here does what is done for other platforms (e.g., darwin needs a different name ending, just not a different mode to go with it) - will "fix" all "performance" related issues for AIX re: calling ldconfig (which is only available in extremely rare situations - again I have never seen it - because, by default, even gcc is using AIX ld, not GNU ld)

I hope saying "please" helps. Without it, the AIX implementation is non-existant. The linux code is called for AIX because that is the last else: block, not because the code is specific to "posix".

so - PLEASE - pretty please!

Michael

p.s. And I shall look at the mercurial pages - and I hope have it working.

---------- nosy: +martin.panter title: ctypes and AIX - also for 2.7.X (and later) -> ctypes library loading and AIX - also for 2.7.X (and later)

Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue27435\

The more common TESTED "behavior" issue is simply:

find_library("c") always returns None while it should be returning either libc.a(shr.o) in 32-bit mode and libc.a(shr_64.o) in 64-bit mode. If that gets corrected, then adding RTLD_MEMBER it's numerical equivalent is also needed.

michael@x071:[/home/michael]grep RTLD_MEMBER /usr/include/.h /usr/include/dlfcn.h:#define RTLD_MEMBER 0x00040000 / Module name may indicate

I mention find_library("c") specifically as that is used as a core test in both Lib/ctypes/utils.py as well as the following tests in Lib/ctypes/test/*.py:

test_callback.py, test_error.py and test_loading.py

How can a test that is always returned None really be testing anything. In any case, what these test are "testing" something very different on AIX compared to Linux (and perhaps Solaris and OS/X).

So, how is the current behavior not a "bug" (that, for ages, has either been ignored and/or not understood - and I go more for the later, as there have been a few issues re: performance concerns because ldconfig is (generally) not available on AIX. Again, the net result of find_library() is for default installs - to return None even when a real value should be returned.

On 11-Aug-16 17:10, Michael Felt wrote:

Martin Panter added the comment: > >For 2.7, adding the automatic RTLD_MEMBER mode does not seem like a bug fix to me. Currently, I understand this code could load two separate libraries: >

The ctypes tests all seem to be protected with code that checks for None, and explicitly skip the test in that case. The skip message should be visible when you run the test in verbose mode.

We could avoid skipping these tests in 2.7 by adding a special case for AIX that loads the library via CDLL("libc.a(shr[_64].o)", RTLD_MEMBER). Like an extension of the special cases already at \https://hg.python.org/cpython/file/v2.7.12/Lib/ctypes/test/test_loading.py#l8\, but we have to set the RTLD_ flag as well as the library name.

On 22-Aug-16 04:04, Martin Panter wrote:

Martin Panter added the comment:

The ctypes tests all seem to be protected with code that checks for None, and explicitly skip the test in that case. The skip message should be visible when you run the test in verbose mode. So at least test/test_something.py run cleanly - but why accept code that always returns None - for find_library().

Rather than "hard-code" proper strings in a test - shouldn't the test be demonstrating the "failure" of find_library() to return a suitable response - when it should not be None.

For example, on AIX the IBM provided /usr/lib/libm.a only has static members. So, what has always been wrong (and noone noticed, or is it noone cared) the "built-in" test in util.py has, and does, always fail:

root@x064:[/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes]../../python 
util.py
None
None
None
Traceback (most recent call last):
   File "util.py", line 271, in <module>
     test()
   File "util.py", line 266, in test
     print cdll.LoadLibrary("libm.so")
   File 
"/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes/__init__.py", line 
440, in LoadLibrary
     return self._dlltype(name)
   File 
"/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes/__init__.py", line 
362, in __init__
     self._handle = _dlopen(self._name, mode)
OSError:        0509-022 Cannot load module .
         0509-026 System error: A file or directory in the path name 
does not exist.

The documentation says:

The exact functionality is system dependent.

Are you saying the documentation should be updated to read: On AIX, find_library tries to use the Linux code which generally fails and will return None.

And the closing bit: If wrapping a shared library with |ctypes| \https://docs.python.org/2/library/ctypes.html#module-ctypes\, it /may/ be better to determine the shared library name at development time, and hardcode that into the wrapper module instead of using |find_library()| to locate the library at runtime (so forget about developing code for multiple platforms - find_library() cannot give you that support. p.s. On AIX, you must use a hard-coded name to ctypes.CDLL() - and also add an extra flag - because even though we said "the exact functionality is system dependent" we do not permit new systems to add corrections.

I apologize that my frustration is more than peeking through. But I am frustrated. find_library(), unless I completely misunderstand the documentation, is a helper function to releave the programmer of needing to know the specifics of multiple platforms. My frustration is heightened by what I read about the issues with "uuid" - which also calls find_library() - so again, on AIX it always returns None -- a) find_library is broken; b) there is no libuuid.so or libuuid.a on AIX (and c) 4 of the 5 programs also called do not exist on AIX, and the one that does generates a different string than is compared to).

I understand - AIX has not been a major (user) platform for python. I am amazed that noone from IBM (AIX) development has never shown more interest. Disclaimer: yes, I work for IBM - but am not a developer. Customers do ask me about "why is python so old, why does it not work with XXX" - and only because I am inclined to "get to the bottom" did I even start on this.

If the root cause is: "IBM showed no interest X years ago and we, python, have no interest now in IBM or AIX" then please just say so - and I shall do other things with my time.

In closing - I thought python was multi-platform and I had hoped to be of some help in improving python integration with the platform AIX. Please, one way or the other, help me lessen my frustration.

Thank you for your time and patience with my rant!

We could avoid skipping these tests in 2.7 by adding a special case for AIX that loads the library via CDLL("libc.a(shr[_64].o)", RTLD_MEMBER). Like an extension of the special cases already at \https://hg.python.org/cpython/file/v2.7.12/Lib/ctypes/test/test_loading.py#l8\, but we have to set the RTLD_ flag as well as the library name.

----------

Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue27435\

I had understood that changing find_library() would only be useful in combination with the automatic RTDL_MEMBER detection. If you want to mention lack of support for AIX in the documentation, that is okay by me. If you want to propose an alternative findlibrary() implementation _without affecting the CDLL behaviour, that may be okay too.

For the libuuid problem, for 2.7 I would suggest adding an explicit call to load libc.a(. . .) or whatever is appropriate.

Personally I don’t have a specific interest in AIX, but I am happy to commit patches for AIX if I think they are sensible. I think I mentioned before in the other bug that your changes seemed okay for the next Python version. The beta deadline for 3.6 is in a few weeks (PEP-494), although after that there is still 3.7.

On 23-Aug-16 02:01, Martin Panter wrote:

Martin Panter added the comment: Thank you for your reply!

I had understood that changing find_library() would only be useful in combination with the automatic RTDL_MEMBER detection. Adding the RTLD_MEMBER to the mode would be necessary for support of native archives. Without it, dlopen() will not work. If I understand your comments correctly a python programmer could change all blocks such as: (please forgive syntax errors, if I make one, I may miss a : or the indent might not be straight)

Now: xxxdl = cdll.LoadLibrary(find_library("xxx"))

Patched: if sys.platform.startswith("aix"): dlname = find_library("xxx") if dlname and dlname.endswith(")") and dlname.rfind(".a(") > 0: from _ctypes import RTLD_LOCAL, RTLD_GLOBAL mode = RTLD_LOCAL | 0x00040000 xxxdl = cdll.LoadLibrary(dlname, mode) else if dlname: xxxld = cdll.LoadLibrary(dlname) else: xxxdl = cdll.LoadLibrary(find_library("xxx"))

If you want to mention lack of support for AIX in the documentation, that is okay by me. If you want to propose an alternative findlibrary() implementation _without affecting the CDLL behaviour, that may be okay too. I certainly do not want to document lack of support - I would prefer to see belated support for AIX. If the code above is all that is acceptable for 2.7 then I am still not as happy as I could be - I would rather not require all programmers that are still active in Python2 to add a new function to replace the default find_library() - as I think rewriting it as a private function is more secure than finding and changing every call to ctypes.find_library currently in "my" code. And, of course, the other change would be everywhere where I now import ctypes.util would be to import my own function (that would be the only place I imports ctypes. In short, what I hope for is that the "Now:" codeblock above will work asis.

For the libuuid problem, for 2.7 I would suggest adding an explicit call to load libc.a(. . .) or whatever is appropriate. Again, without also adding the constant 0x00040000 adding an explicit call to "libc.a(shr.o)" (32-bit mode) or "libc.a(shr_64.o)" is pointless. And, if you would permit the constant 0x00040000 in Lib/uuid.py to make those calls work, then why not let it be in Lib/ctypes/init.py ?

Personally I don’t have a specific interest in AIX, but I am happy to commit patches for AIX if I think they are sensible. I think I mentioned before in the other bug that your changes seemed okay for the next Python version. If I recall, I submitted patches but not processed via Mercurial - not had the time to learn that.

I see (now) that you have processed them, and set them for review - many thanks.

I do not have any specific interest in python. My interest is supporting people who want to use python on AIX. And as PEP-373 says Python is supported into/until 2020 - it seems reasonable to correct an omission. A computer does not care whether the block is "Now:" or "Patched:". But I think most python developers would prefer not having to patch all their code. Rather, I fear they will say "we cannot/do not support AIX" because ... (some politically correct answer to not say AIX support is broken).

The beta deadline for 3.6 is in a few weeks (PEP-494), although after that there is still 3.7. Again, I submitted something. It probably needs changes. I'll check the review. What else is needed for that to be accepted?

If it misses 3.6, then I will have some soul searching to do.

----------

Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue27435\

The "core" changes in this patch are very simple - in parallel with the way find_library() is managed in util.py

a) __init__.py:
   add an additional mode bit for call to dlopen() (in __init__.py)
diff -u src/Python-2.7.12/Lib/ctypes/__init__.py python-2.7.12.2/Lib/ctypes/__init__.py
--- src/Python-2.7.12/Lib/ctypes/__init__.py    2016-06-25 21:49:30 +0000
+++ python-2.7.12.2/Lib/ctypes/__init__.py      2016-09-28 12:55:15 +0000
@@ -352,6 +352,16 @@
             flags |= _FUNCFLAG_USE_ERRNO
         if use_last_error:
             flags |= _FUNCFLAG_USE_LASTERROR
+        if _sys.platform.startswith("aix"):
+            """When the name contains ".a(" and ends with ")",
+               asin "libFOO.a(libFOO.so)" this is taken to be an
+               archive(member) syntax for dlopen(), and the mode is adjusted.
+               Otherwise, name is presented to dlopen() as a file argument.
+            """
+            # from _ctypes import RTLD_NOW - not until Python3.7
+            if name and name.endswith(")") and ".a(" in name:
+                RTLD_MEMBER = 0x00040000
+                mode |= RTLD_MEMBER

         class _FuncPtr(_CFuncPtr):
             _flags_ = flags

b) util.py: determine that is is not generic posix and call and import and execute an external .py file c) some prettier (optional!) print statements in self-test

diff -u src/Python-2.7.12/Lib/ctypes/util.py python-2.7.12.2/Lib/ctypes/util.py --- src/Python-2.7.12/Lib/ctypes/util.py 2016-06-25 21:49:30 +0000 +++ python-2.7.12.2/Lib/ctypes/util.py 2016-09-28 08:41:57 +0000 @@ -81,6 +81,14 @@ continue return None

+if sys.platform.startswith("aix"):
+    # find .so members in .a files
+    # using dump loader header information + sys.
+    import ctypes._aix as aix
+
+    def find_library(name):
+        return aix.find_library(name)
+
 elif os.name == "posix":
     # Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump
     import re, tempfile, errno
@@ -247,10 +255,11 @@
         print find_library("msvcrt")

     if os.name == "posix":
-        # find and load_version
-        print find_library("m")
-        print find_library("c")
-        print find_library("bz2")
+        # find
+        print("m\t:: %s" % find_library("m"))
+        print("c\t:: %s" % find_library("c"))
+        print("bz2\t:: %s" % find_library("bz2"))
+        print("crypt\t:: %s" % find_library("crypt"))

         # getattr
 ##        print cdll.m
@@ -262,6 +271,12 @@
             print cdll.LoadLibrary("libcrypto.dylib")
             print cdll.LoadLibrary("libSystem.dylib")
             print cdll.LoadLibrary("System.framework/System")
+        elif sys.platform.startswith("aix"):
+            print("crypto\t:: %s" % cdll.LoadLibrary(find_library("crypto")))
+            if (sys.maxsize < 2**32):
+                print("c\t:: %s" % cdll.LoadLibrary("libc.a(shr.o)"))
+            else:
+                print("c\t:: %s" % cdll.LoadLibrary("libc.a(shr_64.o)"))
         else:
             print cdll.LoadLibrary("libm.so")
             print cdll.LoadLibrary("libcrypt.so")

d) in test_loading.py say test is failed if find_library does not find at least one library

diff -u src/Python-2.7.12/Lib/ctypes/test/test_loading.py python-2.7.12.2/Lib/ctypes/test/test_loading.py
--- src/Python-2.7.12/Lib/ctypes/test/test_loading.py   2016-06-25 21:49:30 +0000
+++ python-2.7.12.2/Lib/ctypes/test/test_loading.py     2016-09-28 12:43:46 +0000
@@ -11,6 +11,11 @@
     libc_name = "coredll"
 elif sys.platform == "cygwin":
     libc_name = "cygwin1.dll"
+elif sys.platform.startswith("aix"):
+    if (sys.maxsize < 2**32):
+        libc_name = "libc.a(shr.o)"
+    else:
+        libc_name = "libc.a(shr_64.o)"
 else:
     libc_name = find_library("c")

@@ -38,11 +43,16 @@ self.assertRaises(OSError, cdll.LoadLibrary, self.unknowndll)

     def test_find(self):
+        # to track that at least one call to find_library() found something
+        found = False
         for name in ("c", "m"):
             lib = find_library(name)
             if lib:
+                found = True
                 cdll.LoadLibrary(lib)
                 CDLL(lib)
+        # test is FAILED if nothing was found
+        self.assertTrue(found)

 @unittest.skipUnless(os.name in ("nt", "ce"),
                      'test specific to Windows (NT/CE)')

Finally - there is the additional file: _aix.py - submitted as an attachment - and a patch with them all

Considering that python-2.7.13 is posed for a release - I ask again, considering that python-2.7.12 (and I expect a few earlier ones) can load "AIX" .a type shared libraries - if you know how to supply the name -

that my patch be included in python-2.7.13.

michael@x071:[/data/prj/python]-2.7.12.0/Lib/ctypes/util.py                    <
--- python-2.7.12/Lib/ctypes/util.py    2016-06-25 21:49:30 +0000
+++ python-2.7.12.0/Lib/ctypes/util.py  2016-12-02 10:12:01 +0000
@@ -262,6 +262,10 @@
             print cdll.LoadLibrary("libcrypto.dylib")
             print cdll.LoadLibrary("libSystem.dylib")
             print cdll.LoadLibrary("System.framework/System")
+        if sys.platform == "aix5":
+            from ctypes import CDLL
+            RTLD_MEMBER =  0x00040000
+            print CDLL("libc.a(shr_64.o)", RTLD_MEMBER)
         else:
             print cdll.LoadLibrary("libm.so")
             print cdll.LoadLibrary("libcrypt.so")

And the result is:

root@x064:[/data/prj/python/python-2.7.12.0/Lib/ctypes]../../python util.py None None None \<CDLL 'libc.a(shr_64.o)', handle e at 70000000013ae10>

In, other words, as is, find_library() will always return None for standard libraries (will only return a value if "root" has extracted and renamed a shared library member as a file) - while, if you know the magic that find_library cannot return - you can load such a library.

IMVHO (in my Very humble opinion) - a bug that has been too long unnoticed - but can be repaired.

There is no additional ability being added - it is already there. Just making it more accessible and legible for python programmers on AIX.

If the patch-code needs more cleanup, improvement, etc. - fine. But do not lock out an easy interface to such a basic function.

regards, Michael

Again, I would like to draw attention to this issue - BECAUSE -

it is possible to use ctypes.CDLL() ASIS, on AIX - however, ctypes.util.find_library() always seems to fail (i.e., returns None even when a .so file exists)

When the library names are hard-coded, rather than found by find_library() everything seems to work fine.

In short, I hope this will finally be recognized as a bug (ctypes.util.find_library()).

Ideally, RTLD_MEMBER logic would be added to the Lib/ctypes/init.py so that CDLL does not have to have RTLD_MEMBER added manually as it does now.

(I have not yet figured out how to use LibraryLoader() properly, it seems to never complain, or behave the same as ctypes.CDLL(None) which returns the "python process" namespace - if I understand that correctly.

p.s. a) I have a newer patch; b) PEP-8, etc. pickyness is encouraged (i.e., expect multiple iterations before a patch is accepted - I do not mind learning PEP-8 et al.);

imho - nothing new is being added to what python2 (and python3!) already can do. However, a fix to make it easier and clearer to use (there must be a PEP for that case as well) - is warranted.

++++++++ Preparation: a) the contents of the archive (example) root@x064:[/data/prj/python/python-2.7.12.0]ar -Xany tv /usr/lib/libcurses.a rwxr-xr-x 300/300 499495 Sep 26 22:35 2007 shr42.o rwxr-xr-x 300/300 321459 Sep 26 22:04 2007 shr4.o rwxr-xr-x 300/300 197552 Sep 26 22:04 2007 shr.o rwxr-xr-x 300/300 591888 Sep 26 22:35 2007 shr42_64.o

b) create a empty .so file - MAYBE this will be found by the so-called "posix" routine in ctypes.util.find_library

root@x064:[/data/prj/python/python2-2.7.13]touch /usr/lib/libcurses.so

c) A test program import sys import ctypes from ctypes import util

if sys.platform[:3] == "aix":
    print ctypes.util.find_library("curses")
    RTLD_MEMBER =  0x00040000
    if (sys.maxsize < 2**32):
        print 32
        print ctypes.CDLL("libcurses.a(shr.o)", RTLD_MEMBER)
    else:
        print 64
        print ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER)

Testing: a) notice "None" as value printed by find_library b) notice that ctypes.CDLL can load the library when RTLD_MEMBER isd added root@x064:[/data/prj/python/python2-2.7.13]./python ../python2-2.7.13/test*.py None 64 \<CDLL 'libcurses.a(shr42_64.o)', handle 10 at 7000000001280f0>

Changes: -- since /usr/lib/libcurses.so is an empty file MAYBE - that is why the result was None -- extract the 64-bit member and rename as /usr/lib/libcurses.so -- test both find_library (still says None!) and CDLL of the extracted member -- as it also loads, CDLL, i.e., dlopen(), works without RTLD_MEMBER (as it should) -- there does not appear top be anything wrong with ctypes.CDLL, but only with the 'utils' included

root@x064:[/data/prj/python/python2-2.7.13]ar x /usr/lib/libcurses.a shr42_64.o root@x064:[/data/prj/python/python2-2.7.13]mv shr42_64.o /usr/lib/libcurses.so

import sys
import ctypes
from ctypes import util

if sys.platform[:3] == "aix":
    print ctypes.util.find_library("curses")
    RTLD_MEMBER =  0x00040000
    if (sys.maxsize < 2**32):
        print 32
        print ctypes.CDLL("libcurses.a(shr.o)", RTLD_MEMBER)
    else:
        print 64
        print ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER)
        print ctypes.CDLL("libcurses.so")

root@x064:[/data/prj/python/python2-2.7.13]./python ../python2-2.7.13/test*.py None 64 \<CDLL 'libcurses.a(shr42_64.o)', handle 10 at 700000000128128> \<CDLL 'libcurses.so', handle 11 at 700000000128128>

Verification of (expected) behavior: a) CDLL in both formats returns the same function pointer for 'addch' b) LibraryLoader always returns something, even when it is nonsense c) CDLL - fails - aka Traceback - when an argument cannot be loaded

from ctypes import util

if sys.platform[:3] == "aix":
    print ctypes.util.find_library("curses")
    RTLD_MEMBER =  0x00040000
    if (sys.maxsize < 2**32):
        print 32
        print ctypes.CDLL("libcurses.a(shr.o)", RTLD_MEMBER)
    else:
        print 64
        print ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER)
        lib = ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER)
        if hasattr(lib, 'addch'):
            print "addch found"
            print lib.addch
        else:
            print "addch not found"
        print ctypes.CDLL(None)
        lib = ctypes.CDLL(None)
        if hasattr(lib, 'addch'):
            print "addch found"
            print lib.addch
        else:
            print "addch not found"
        print ctypes.CDLL("libcurses.so")
        lib = ctypes.CDLL("libcurses.so")
        if hasattr(lib, 'addch'):
            print "addch found"
            print lib.addch
        else:
            print "addch not found"
        print "LibraryLoader tests"
        print ctypes.LibraryLoader("libcurses.so")
        lib = ctypes.LibraryLoader("libcurses.so")
        print ctypes.LibraryLoader(None)
        print ctypes.LibraryLoader("libcurses.a(shr42_64.o)")
        print ctypes.LibraryLoader("libc.a(shr_64.o)")
        print "LibraryLoader XXX tests show LibraryLoader masks unloadable libraries"
        print ctypes.LibraryLoader("xxx.so")
        print ctypes.CDLL("xxx.so")

returns: (python2-2.7.12 and python2-2.7.13 return the same value)

root@x064:[/data/prj/python/python-2.7.12.0]./python ../python2-2.7.13/test*.py
None
64
<CDLL 'libcurses.a(shr42_64.o)', handle e at 700000000135a58>
addch found
<_FuncPtr object at 0x700000000174108>
<CDLL 'None', handle 10 at 7000000001354a8>
addch not found
<CDLL 'libcurses.so', handle 12 at 7000000001358d0>
addch found
<_FuncPtr object at 0x700000000174048>
LibraryLoader tests
<ctypes.LibraryLoader object at 0x7000000001354a8>
<ctypes.LibraryLoader object at 0x700000000135470>
<ctypes.LibraryLoader object at 0x700000000135470>
<ctypes.LibraryLoader object at 0x700000000135470>
LibraryLoader XXX tests show LibraryLoader masks unloadable libraries
<ctypes.LibraryLoader object at 0x700000000135470>
Traceback (most recent call last):
  File "../python2-2.7.13/test_ctypes.py", line 42, in <module>
    print ctypes.CDLL("xxx.so")
  File "/data/prj/python/python-2.7.12.0/Lib/ctypes/__init__.py", line 362, in __init__
    self._handle = _dlopen(self._name, mode)
OSError:        0509-022 Cannot load module .
        0509-026 System error: A file or directory in the path name does not exist.

root@x064:[/data/prj/python/python-2.7.12.0]cd ../py*13

root@x064:[/data/prj/python/python2-2.7.13]./python ../python2-2.7.13/test*.py
None
64
<CDLL 'libcurses.a(shr42_64.o)', handle 10 at 700000000127320>
addch found
<_FuncPtr object at 0x700000000158288>
<CDLL 'None', handle 12 at 700000000127710>
addch not found
<CDLL 'libcurses.so', handle 14 at 700000000127898>
addch found
<_FuncPtr object at 0x7000000001581c8>
LibraryLoader tests
<ctypes.LibraryLoader object at 0x700000000127908>
<ctypes.LibraryLoader object at 0x700000000127978>
<ctypes.LibraryLoader object at 0x700000000127978>
<ctypes.LibraryLoader object at 0x700000000127978>
LibraryLoader XXX tests show LibraryLoader masks unloadable libraries
<ctypes.LibraryLoader object at 0x700000000127978>
Traceback (most recent call last):
  File "../python2-2.7.13/test_ctypes.py", line 42, in <module>
    print ctypes.CDLL("xxx.so")
  File "/data/prj/python/src/python2-2.7.13/Lib/ctypes/__init__.py", line 362, in __init__
    self._handle = _dlopen(self._name, mode)
OSError:        0509-022 Cannot load module .
        0509-026 System error: A file or directory in the path name does not exist.
root@x064:[/data/prj/python/python2-2.7.13]

Feels like it is up to me to shed some light on various topics here:

AIX defines (static) "Object" files and "Shared Object" files. Both can (but do not need to) be members of an "Archive Library".

When talking about (dynamically loading) a "shared library" on non-AIX systems, the correct wording for AIX is (dynamically loading) a "Shared Object" - independent of whether it is a standalone file or an archive member.

As you already agreed on, ctypes.util.find_library() indeed should return the location of the Shared Object - as either "/path/to/file" or "/path/to/file(member)". And ctypes.CDLL() should add the RTLD_MEMBER flag to _ctypes.dlopen() if there is the "(member)" part.

However, that does not necessarily mean the RTLD_MEMBER value needs to be available through the ctypes API. Instead, RTLD_MEMBER feels more appropriate for the _ctypes API - and on AIX only.

Anyway: Unfortunately, there is no concept of embedding something like ELF's DT_SONAME tag into the Shared Object. The very (PATH,BASE,MEMBER) value as (specified to and) discovered by the linker is recorded into the just-linked executable (or Shared Object). This implies that the runtime loader does search for the very same filename (and member eventually) as the linker (at linktime).

However, to still get some kind of shared library versioning, multiple versions of one Shared Object are put into one single Archive Library, where the "old" versions get the LOADONLY flag set (via 'strip -e') - so executables linked against an old version still can load their "old" Shared Object, while the linker discovers the "current" version only.

But this "single-filename" based shared library versioning is a huge burden for packages managers - either human or software, even when they maintain their packages in a private prefix (like /opt/freeware, /usr/local and similar). Neither can they put their "current" Shared Object's version into the single (system's) Archive Library, nor can they mark existing Shared Objects as LOADONLY. On the other hand, they also do not want to pull all the system versions into their own Archive Library.

So for package managers it is crucial to have "multi-filename" based shared library versioning instead.

But there is help: AIX also provides (plain text) "Import Files": They contain a list of symbols for the linker, and a (PATH,BASE,MEMBER) tuple naming the Shared Object that provides these symbols at runtime.

So yes, Import Files can be used for "multi-filename" based shared library versioning.

Thus, libtool now offers the "--with-aix-soname=svr4" configure flag, where "libNAME.so.X" really is created as an Archive Library, containing the Shared Object "shr.o" with the LOADONLY flag set, and the Import File "shr.imp" referring to "libNAME.so.X(shr.o)" for runtime. Combined with the symlink "libNAME.so", for the "-lNAME" argument the linker discovers the Import File "libNAME.so(shr.imp)" now, while recording "libNAME.so.X(shr.o)" into the executable for the runtime loader. Note that for 64bit the Shared Object and Import File is named "shr_64.o" and "shr_64.imp", respectively.

While libtool's "--with-aix-soname=svr4" variant creates "libNAME.a" from static Objects only, both libtool's "--with-aix-soname=aix" and "--with-aix-soname=both" - for backwards compatibility - create "libNAME.a" for traditional "single-filename" based versioning: Without any static Object - as even Shared Objects can be linked statically(!). But to statically link the Shared Object (as an archive member), neither the LOADONLY flag must be set, nor an Import File must be found (as Import Files cannot serve for static linking).

And "--with-aix-soname=aix", libtool still creates standalone "libNAME.so.X.Y.Z" along the (versioning-wise useless) symlinks.

So it is up to the package manager which "aix-soname" variant to choose within its prefix: For example, in Gentoo Prefix I'm using "--with-aix-soname=svr4" only.

But still, how to get ctypes.find_library() working - ideally for each variant, is another story. Right now it does not work for any variant, but I guess that search algorithm should follow how the linker discovers the (PATH,BASE,MEMBER) values to write into just-linked executables, combined with how the runtime loader finds the Shared Object to actually load.

On 31/01/2017 20:22, Michael Haubenwallner wrote:

Michael Haubenwallner added the comment:

Feels like it is up to me to shed some light on various topics here: Many thanks!

AIX defines (static) "Object" files and "Shared Object" files. Both can (but do not need to) be members of an "Archive Library".

When talking about (dynamically loading) a "shared library" on non-AIX systems, the correct wording for AIX is (dynamically loading) a "Shared Object" - independent of whether it is a standalone file or an archive member.

As you already agreed on, ctypes.util.find_library() indeed should return the location of the Shared Object - as either "/path/to/file" or "/path/to/file(member)". And ctypes.CDLL() should add the RTLD_MEMBER flag to _ctypes.dlopen() if there is the "(member)" part.

However, that does not necessarily mean the RTLD_MEMBER value needs to be available through the ctypes API. Instead, RTLD_MEMBER feels more appropriate for the _ctypes API - and on AIX only.

Anyway: Unfortunately, there is no concept of embedding something like ELF's DT_SONAME tag into the Shared Object. The very (PATH,BASE,MEMBER) value as (specified to and) discovered by the linker is recorded into the just-linked executable (or Shared Object). This implies that the runtime loader does search for the very same filename (and member eventually) as the linker (at linktime). I assume this is why there are many systems besides AIX that do not support/use DT_SONAME. At least I see many references to "Shared Objects" libFOO.so.X.Y.Z, libFOO.so.X.Y, libFOO.so.X and libFOO.so (with the latter three being symbolic links to the first).

However, to still get some kind of shared library versioning, multiple versions of one Shared Object are put into one single Archive Library, where the "old" versions get the LOADONLY flag set (via 'strip -e') - so executables linked against an old version still can load their "old" Shared Object, while the linker discovers the "current" version only.

But this "single-filename" based shared library versioning is a huge burden for packages managers - either human or software, even when they maintain their packages in a private prefix (like /opt/freeware, /usr/local and similar). Neither can they put their "current" Shared Object's version into the single (system's) Archive Library, nor can they mark existing Shared Objects as LOADONLY. On the other hand, they also do not want to pull all the system versions into their own Archive Library. Another issue is support for what I believe MacOS calls "fat" objects - that support both 32-bit and 64-bit applications - rather than /XXX/lib for 32-bit objects and /XXX/lib/lib64 or /XXX/lib64 for 64-bit objects.

So for package managers it is crucial to have "multi-filename" based shared library versioning instead.

But there is help: AIX also provides (plain text) "Import Files": They contain a list of symbols for the linker, and a (PATH,BASE,MEMBER) tuple naming the Shared Object that provides these symbols at runtime. a) Yes to above b) One of the difficulties I faced is trying to guess what version -lFOO should find when there is more than one version available. Applications that are already linked do not look for a latest version - they know the member name that the linker found initially. ctypes.utils.find_library() does not provide a way to say which version should be found. Each implementation has it's way to find a version (e.g., a search of ldconfig -p output for a best guess) So yes, Import Files can be used for "multi-filename" based shared library versioning.

Thus, libtool now offers the "--with-aix-soname=svr4" configure flag, where "libNAME.so.X" really is created as an Archive Library, containing the Shared Object "shr.o" with the LOADONLY flag set, and the Import File "shr.imp" referring to "libNAME.so.X(shr.o)" for runtime. Combined with the symlink "libNAME.so", for the "-lNAME" argument the linker discovers the Import File "libNAME.so(shr.imp)" now, while recording "libNAME.so.X(shr.o)" into the executable for the runtime loader. Note that for 64bit the Shared Object and Import File is named "shr_64.o" and "shr_64.imp", respectively.

While libtool's "--with-aix-soname=svr4" variant creates "libNAME.a" from static Objects only, both libtool's "--with-aix-soname=aix" and "--with-aix-soname=both" - for backwards compatibility - create "libNAME.a" for traditional "single-filename" based versioning: Without any static Object - as even Shared Objects can be linked statically(!). But to statically link the Shared Object (as an archive member), neither the LOADONLY flag must be set, nor an Import File must be found (as Import Files cannot serve for static linking).

And "--with-aix-soname=aix", libtool still creates standalone "libNAME.so.X.Y.Z" along the (versioning-wise useless) symlinks.

So it is up to the package manager which "aix-soname" variant to choose within its prefix: For example, in Gentoo Prefix I'm using "--with-aix-soname=svr4" only.

But still, how to get ctypes.find_library() working - ideally for each variant, is another story. Right now it does not work for any variant, Do you mean all systems, or specific to AIX - I am assuming you mean AIX. but I guess that search algorithm should follow how the linker discovers the (PATH,BASE,MEMBER) values to I am not a tool builder. My comments are based on observations and experience from when I was a developer 25+ years ago. The AIX linker is not interested in the member name - it seems to go through the PATH/libBASE.a looking for the first object it can find to resolve a symbol. The name of the object it finds becomes the MEMBER it records in it's internal table of where to look later when the application runs.

This is why (on AIX) when a new archive is installed - even though there is a MEMBER that could resolve a symbol, if the MEMBER name has changed the rtld activity will fail. This is where (at least on AIX) using the LDONLY flag is important. The "old" MEMBER is there for "old" applications, but "new" applications will use a different object to resolve symbols. I would say the linker seraches for something to resolve a symbol (get it defined) while the rtld (run-time loader) locates a known BASE,MEMBER pair based on the internal "blibpath" and/or an optional environmental variable (LD_LIB_PATH, LD_LIBRARY - must check exact names - in any case AIX dlopen() knows, better recognizes two environmental variables, as a list of PATHs to follow to locate the BASE(MEMBER) pair.

write into just-linked executables, combined with how the runtime loader finds the Shared Object to actually load. I worked on a patch - to do all that - taking into consideration the way libtool names .so files/members and then looking into/at "legacy" aka IBM dev ways they did things before the libtool model was so prominent.

My algorithm - attempts to solve the (PATH, BASE, MEMBER) problem as "dynamically" as possible. PATH and BASE are fairly straight forward - but MEMBER is clearly more complex.

PATH: start by looking at the python executable - and looking at it's "blibpath" - and using that as the default colon separated list of PATHs to search for BASE.a archive. Once a BASE.a file is found it is examined for a MEMBER. If all PATH/BASE.a do not find a potential MEMBER then the PATHs are examined again for PATH/BASE.so. When a .so file is found that is returned - versioning must be accomplished via a symbolic link to a versioned library.

The program "dump -H" provides this information for both executables and archive (aka BASE) members.

Starting from the "blibpath" values in the executable mean a cpython packager can add a specific PATH by adding it to LDFLAGS="-L/my/special/libdir:$LDFLAGS". Note that AIX archives also have their own "blibpath" - so libraries dynamically loaded may also follow additional paths that the executable is not aware of (nor need to be).

So - once the PATHS are determined the system is examined looking for ${PATH}/BASE.a. If a target BASE.a is found, it is examined for a MEMBER is set to BASE.so (now used a MEMBER.so) . If MEMBER.so is not found then look for the "highest X[.Y[.Z]] aka MEMBER.so.X[.Y[.Z]] name. If that is not found check AIX legacy names (mainly shr.o or shr_64.o, although there are also certain key libraries that have additional variations (boring)).

Again, if PATH, BASE, MEMBER is not located as a .a archive - look for libFOO.so in all the PATH directories known to the executable.

I hope above is clear enough to continue the discussion. Thanks again for the discussion.

Michael

----------

Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue27435\

On 02/03/2017 09:52 PM, Michael Felt wrote:

> Anyway: > Unfortunately, there is no concept of embedding something like ELF's DT_SONAME tag into the Shared Object. > The very (PATH,BASE,MEMBER) value as (specified to and) discovered by the linker is recorded into the just-linked executable (or Shared Object). > This implies that the runtime loader does search for the very same filename (and member eventually) as the linker (at linktime).

I assume this is why there are many systems besides AIX that do not support/use DT_SONAME.

Except for Windows, I'm not sure which "many systems besides AIX" you're talking here about, that "do not use/support DT_SONAME".

At least I see many references to "Shared Objects" libFOO.so.X.Y.Z, libFOO.so.X.Y, libFOO.so.X and libFOO.so (with the latter three being symbolic links to the first).

When a system happens to find these symlinks useful, then it actually _does_ support embedding DT_SONAME (or something similar) into its binary file format.

Another issue is support for what I believe MacOS calls "fat" objects - that support both 32-bit and 64-bit applications - rather than /XXX/lib for 32-bit objects and /XXX/lib/lib64 or /XXX/lib64 for 64-bit objects.

Yes, the AIX Archive Libraries supporting different bitwidths for members is quite similar to MacOS fat objects. However - although related, the creation of "fat" AIX archives is a different topic. But yes, Python ctypes.find_library+dlopen should know how to deal with them.

b) One of the difficulties I faced is trying to guess what version -lFOO should find when there is more than one version available.

Exactly. There is an idea below (the symbol->member map).

> But still, how to get ctypes.find_library() working - ideally for each variant, is another story. Right now it does not work for any variant, Do you mean all systems, or specific to AIX - I am assuming you mean AIX.

Yes - find_library currently does not work for any variant on *AIX*.

> but I guess that search algorithm should follow how the linker discovers the (PATH,BASE,MEMBER) values to

I am not a tool builder. My comments are based on observations and experience from when I was a developer 25+ years ago. The AIX linker is not interested in the member name - it seems to go through the PATH/libBASE.a looking for the first object it can find to resolve a symbol. The name of the object it finds becomes the MEMBER it records in it's internal table of where to look later when the application runs.

Exactly.

> write into just-linked executables, combined with how the runtime loader finds the Shared Object to actually load.

I worked on a patch - to do all that - taking into consideration the way libtool names .so files/members and then looking into/at "legacy" aka IBM dev ways they did things before the libtool model was so prominent.

My algorithm - attempts to solve the (PATH, BASE, MEMBER) problem as "dynamically" as possible. PATH and BASE are fairly straight forward - but MEMBER is clearly more complex.

PATH: start by looking at the python executable -

As far as I can tell, any executable can actually link against the Python interpreter.

and looking at it's "blibpath" -

There also is the loadquery() subroutine in AIX, see https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.basetrf1/loadquery.htm

loadquery(L_GETLIBPATH) "Returns the library path that was used at process exec time.", which includes both the environment variable LIBPATH (or LD_LIBRARY_PATH if LIBPATH is unset) and the executable's "blibpath" value.

and using that as the default colon separated list of PATHs

Question is if we do want to consider _current_ values of environment variable LIBPATH (or LD_LIBRARY_PATH) in addition to the "library path at process exec time"?

to search for BASE.a archive. Once a BASE.a file is found it is examined for a MEMBER. If all PATH/BASE.a do not find a potential MEMBER then the PATHs are examined again for PATH/BASE.so.

Erm, nope, the AIX linker has a different algorithm (for -lNAME): Iterating over the "library path", the first path entry containing any matching filename (either libNAME.a or libNAME.so) will be used, and no further library path iteration is performed. This one found PATH/filename does have to provide the requested symbol in one way or another.

When a .so file is found that is returned - versioning must be accomplished via a symbolic link to a versioned library.

The linker does not perform such a check, nor does it feel necessary for ctypes.find_library+dlopen as long as it does search similar to the linker.

The program "dump -H" provides this information for both executables and archive (aka BASE) members.

Eventually we might want to avoid spawning the 'dump' program, but implement reading the XCOFF Object File Format within _ctypes module instead. At least AIX does provide the necessary headers: https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.files/XCOFF.htm

Starting from the "blibpath" values in the executable mean a cpython packager can add a specific PATH by adding it to LDFLAGS="-L/my/special/libdir:$LDFLAGS". Note that AIX archives also have their own "blibpath" - so libraries dynamically loaded may also follow additional paths that the executable is not aware of (nor need to be).

There is no need for the ctypes module to search libpaths from other Shared Objects than the main executable (and current env vars).

So - once the PATHS are determined the system is examined looking for ${PATH}/BASE.a. If a target BASE.a is found, it is examined for a MEMBER is set to BASE.so (now used a MEMBER.so) . If MEMBER.so is not found then look for the "highest X[.Y[.Z]] aka MEMBER.so.X[.Y[.Z]] name. If that is not found check AIX legacy names (mainly shr.o or shr_64.o, although there are also certain key libraries that have additional variations (boring)).

When ctypes.dlopen is asked to load an Archive Library (either .a or .so) without a specific member, it probably should not immediately dlopen a specific member, but fetch the list of symbols provided by useable members (either Shared Objects without the F_LOADONLY flag, as well as specified in Import Files), and return the handle to some internal symbol->member map instead.

Then, really loading a shared archive member is done by subsequent ctypes.dlsym - where it becomes clear which archive member to load.

Again, if PATH, BASE, MEMBER is not located as a .a archive - look for libFOO.so in all the PATH directories known to the executable.

Nope, see above - please iterate over a libpath list only _once_, and search for each filename while at one path list entry.

However, I'm not sure yet how to identify if we should search for .a or .so first (before .so and .a, respectively): This depends on whether the current executable actually does use runtime linking or not, but I have no idea yet how to figure that out. But probably we may want to use how the Python interpreter (libpython.a or libpython.so) was built.

Uhm, ultimative solution feels complex already, while still some things to decide...

Let's switch to github-based patches to discuss about: https://github.com/python/cpython/compare/2.7...haubi:bpo-27435/2.7

For the library search path I prefer to use loadquery(L_GETLIBPATH), available via new _ctypes_aix module now, but also used with L_GETINFO in Python/dynload_aix.c already.

First, my apology that I have not responded earlier. I had other things to work on (real life things), customers that had interest in a fix for find_library() have indicated no longer have interest, and also my personal issue - becoming disillusioned with the lack of progress in the discussion.

Clearly, the detail of your comments proves me wrong on that final point. I will be more attentive.

Jumping to your last comment:

Uhm, ultimative solution feels complex already, while still some things to decide...

So, rather than try for perfection in one go - set some priorities.

IMHO (emphasis on the H) - having find_library return something useful, even if not right 100% of the time - is better than wrong 100% of the time. - this concerns find_library().

Your "find" of the routine "loadquery()" may be a big improvement over what I found first. FYI - while you show links to AIX 7.2 documentation I am, as a 'packager' still trying, for the time being, to package on AIX 5.3. I know AIX 5.3 is no longer supported, but as long as something such as RBAC intelligence is not needed I prefer to package only once - and let AIX binary compatibility upwards do it's thing. Back to the H from above: loadquery() is an enhancement - and in previous discussions about my patch, such enhancements were frowned upon (i.e., it seemed that anything that would consider or include LD_LIBRARY_PATH or LIBPATH would not be considered - so even something that would examine the environment (i.e., an exported LD_LIBRARY_PATH or LIBPATH) was not acceptable. However, I concur loadquery() using *L_GETLIBPATH*can be expected to be more efficient (and easier to read?) that my earlier attempts to get this info using "dump -H".

More comments (a few) below.

On 20/02/2017 14:37, Michael Haubenwallner wrote:
> Michael Haubenwallner added the comment:
>
> On 02/03/2017 09:52 PM, Michael Felt wrote:
>>> Anyway:
>>> Unfortunately, there is no concept of embedding something like ELF's DT_SONAME tag into the Shared Object.
>>> The very (PATH,BASE,MEMBER) value as (specified to and) discovered by the linker is recorded into the just-linked executable (or Shared Object).
>>> This implies that the runtime loader does search for the very same filename (and member eventually) as the linker (at linktime).
>> I assume this is why there are many systems besides AIX that do not
>> support/use DT_SONAME.
> Except for Windows, I'm not sure which "many systems besides AIX" you're talking here about, that "do not use/support DT_SONAME".
Clearly, my assumption is wrong. I "grew up" on BSD Unix, not System V, 
and AIX seems (imho) favor BSD in some aspects. I assume (emphasis 
assume!) that soname is not part of the POSIX standard (at least not as 
early as 2005 - standards change).
>> At least I see many references to "Shared
>> Objects" libFOO.so.X.Y.Z, libFOO.so.X.Y, libFOO.so.X and libFOO.so (with
>> the latter three being symbolic links to the first).
> When a system happens to find these symlinks useful, then it actually _does_ support embedding DT_SONAME (or something similar) into its binary file format.
What I have seen on AIX - packaging by others, and from memory what 
libtool is doing, is the following: the file: libfoo.so.X.Y.Z gets 
created, the following symbolic links are created - each pointing at 
libfoo.so.X.Y.Z: libfoo.so.X.Y, libfoo.so.X and libfoo.so. I also see 
the same "logic" in IBM provided archives. I use libssl.a as my guide 
(for 'versioning'), but most of my 'issues' has been with supporting 
backwards compatibility with libintl.a - execpt here they are not 
symbolic links. The same "file" is added to the archive, but with a 
different name - so if a "soname extension" was used (better, found 
during linking), it is used. The order in the archive is important. If 
the generic name is first, then that is the name that will be used 
during linking (and remembered for execution).

root@x064:[/usr/lib]lslpp -L | grep openssl.base openssl.base 1.0.2.1000 C F Open Secure Socket Layer root@x064:[/usr/lib]ar -Xany -tv libssl.a rwxr-xr-x 537912/767508 726474 Oct 18 11:38 2016 libssl.so rwxr-xr-x 537912/767508 726474 Oct 18 11:38 2016 libssl.so.1.0.0 rwxr-xr-x 537912/767508 510610 Oct 18 11:39 2016 libssl.so.0.9.8 rwxr-xr-x 537912/767508 823217 Oct 18 11:39 2016 libssl64.so rwxr-xr-x 537912/767508 823217 Oct 18 11:39 2016 libssl64.so.1.0.0 rwxr-xr-x 537912/767508 577122 Oct 18 11:54 2016 libssl64.so.0.9.8

root@x064:[/usr/lib]lslpp -L aixtools.gnu.gettext.rte Fileset Level State Type Description (Uninstaller) ---------------------------------------------------------------------------- aixtools.gnu.gettext.rte 0.19.8.1 C F built 21-Aug-2016 1821 UTC

root@x064:[/usr/lib]ar -Xany tv libintl.a rwxr-xr-x 0/0 87530 Aug 21 16:45 2016 libintl.so.8 rwxr-xr-x 0/0 79727 Aug 21 18:17 2016 libintl.so.8 rw-r--r-- 0/0 3753 Jun 03 18:05 2017 bindtextdom.o ... rw-r--r-- 0/0 2014 Jun 03 18:05 2017 textdomain.o rwxr-xr-x 0/0 64001 Jun 03 18:05 2017 libintl.so.1

Here, to keep ancient legacy programs - I have copied (and put at the end of the archive the 32-bit libintl.so.1 member that some programs are still using - sadly).

And another example - with libz.a - where, historical moments can be important as well: -- Here I overwrite the member provided by IBM (libz.so.1) - and do not provide libz.so - as this has been the convention - forever.

root@x064:[/usr/lib]ar -Xany -tv libz.a rwxr-xr-x 0/0 174334 Jan 31 12:53 2017 libz.so.1 rwxr-xr-x 0/0 174334 Jan 31 12:53 2017 libz.so.1.2.11 rwxr-xr-x 0/1954 174270 Feb 02 18:55 2017 libz.so.1.2.10 r-xr-xr-x 0/1954 164528 Feb 02 18:55 2017 libz.so.1.2.8 rw-r--r-- 0/0 5565 Jan 31 12:53 2017 adler32.o rw-r--r-- 0/0 13383 Jan 31 12:53 2017 crc32.o ... rw-r--r-- 0/0 10973 Jan 31 12:53 2017 gzwrite.o rwxr-xr-x 0/0 164632 Jan 31 12:55 2017 libz.so.1 rwxr-xr-x 0/1954 164632 Feb 02 18:43 2017 libz.so.1.2.11 rwxr-xr-x 0/1954 164594 Feb 02 18:53 2017 libz.so.1.2.10 r-xr-xr-x 0/1954 156861 Feb 02 18:53 2017 libz.so.1.2.8 rw-r--r-- 0/0 5249 Jan 31 12:55 2017 adler32.o rw-r--r-- 0/0 13880 Jan 31 12:55 2017 crc32.o ...

So, getting back to your comment about it being "complex". Maybe I understand that in details that few have cared to investigate. And I am skipping over the cases where IBM uses the name shr.o as the shared library that needs to be loaded (as find_library("c") should return - as just one example, or shr_64.o - when a 64-bit version of python is packaged) (example from AIX 5.3 TL7 - and I would expect to see the same on AIX 7.2)

root@x064:[/usr/lib]ar -Xany -tv libc.a | grep shr rwxr-xr-x 300/300 4154464 Sep 26 20:00 2007 shr.o rwxr-xr-x 300/300 4516730 Sep 26 20:00 2007 shr_64.o

> Another issue is support for what I believe MacOS calls "fat" objects - > that support both 32-bit and 64-bit applications - rather than /XXX/lib > for 32-bit objects and /XXX/lib/lib64 or /XXX/lib64 for 64-bit objects. Yes, the AIX Archive Libraries supporting different bitwidths for members is quite similar to MacOS fat objects. However - although related, the creation of "fat" AIX archives is a different topic. But yes, Python ctypes.find_library+dlopen should know how to deal with them. I am not asking steps on how to create them - that is my 'private' problem as a packager. I can tell you it entails building the package twice - plus some additional steps.

> b) One of the difficulties I faced is trying to guess what version -lFOO > should find when there is more than one version available. Exactly. There is an idea below (the symbol->member map). I looked at that - and my expectation is: wonderful, but not for 2.7 as it is a different behavior than find_library() anno Python2.7 (or any version Lib/ctypes for that matter) - and should not be limited to AIX, but should deal with all platforms and how they provide multiple versions. I have seen packages that try to resolve that now by doing - potentially, multiple calls to find_library - and supply specific values for libfoo.so.X. That is, I believe there are users (i.e., python application/module developers) that would make use of the added functionality. However, for now I think it is simpler - on AIX at least

to 'assume' that libfoo.so is a symbolic link to libfoo.so.X[[.Y][.Z]] (hope I have the [] correctly and/or you understand my intent) - as that is the 'default' behavior of libtool. And, when file or "member" libfoo.so does not exist to go for the most specific .X.Y.Z available.

>>> But still, how to get ctypes.find_library() working - ideally for each variant, is another story. Right now it does not work for any variant,
>> Do you mean all systems, or specific to AIX - I am assuming you mean AIX.
> Yes - find_library currently does not work for any variant on *AIX*.
So, it is always up to the programmer to find a solution. And, my 
observation is that they create an unmaintable/unmanageable situation by 
extracting a member from an archive into /usr/lib and hard-coding the 
library-name in a call to LoadLibrary(). Of course if the archive they 
extracted from gets updated - their extracted member is not updated. etc.

If they have done enough research (which I have not seen) they could use

instead (part of my would-be 'patch' to Lib/ctypes/util.py) # load if sys.platform == "darwin": print cdll.LoadLibrary("libm.dylib") print cdll.LoadLibrary("libcrypto.dylib") print cdll.LoadLibrary("libSystem.dylib") print cdll.LoadLibrary("System.framework/System") elif sys.platform[:3] == "aix": from ctypes import CDLL RTLD_MEMBER = 0x00040000 print CDLL("libc.a(shr.o)", RTLD_MEMBER) else: print cdll.LoadLibrary("libm.so") print cdll.LoadLibrary("libcrypt.so") print find_library("crypt")

And gives: root@x064:[/data/prj/python/python-2.7.13.0/Lib/ctypes]../../python util.py None None None \<CDLL 'libc.a(shr.o)', handle 10 at 300fe2f0>

So, just want to point out - if findlibrary() was working, there is only a minor change to \_init__.py needed (specifically to add the RTLD_MEMBER to the mode when the filename includes '(' and ends in ')'.

In short, for AIX find_library() is broken. period. It could be fixed so that code written as: cdll.LoadLibrary(find_library("foo")) would work cross platform. (I do not have a mac to test on, but I expect find_library("m") returns 'libm.dylib'. FYI: On AIX, find_library("m") should return None - as there is no shared library in libm.a

>>> but I guess that search algorithm should follow how the linker discovers the (PATH,BASE,MEMBER) values to
>> I am not a tool builder. My comments are based on observations and
>> experience from when I was a developer 25+ years ago. The AIX linker is
>> not interested in the member name - it seems to go through the
>> PATH/libBASE.a looking for the first object it can find to resolve a
>> symbol. The name of the object it finds becomes the MEMBER it records in
>> it's internal table of where to look later when the application runs.
> Exactly.
See above re: my observations re: current practice/behavior

I could go through all of PEP-20 - but one of my goals was to have AIX low-level details masked, so that code "written for any platform" such as:

# leaving out the 'include statements' to show the 'core' objective
solib = None
solibname = find_library("foo")
if solibname:
     solib = cdll.LoadLibrary(solibname)
# optionally, there might also be an else block dealing with the 
'unexpected?' lack of solibname resolution.

would work.

There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea.

I would like to add - I may be American born, I carry a Dutch passport when traveling in EU. ;)

And, while *right* now may be too soon - but remember 'never' is - I hope in the spirit of PEP-20 - too late.

>
>>>    write into just-linked executables, combined with how the runtime loader finds the Shared Object to actually load.
>> I worked on a patch - to do all that - taking into consideration the way
>> libtool names .so files/members and then looking into/at "legacy" aka
>> IBM dev ways they did things before the libtool model was so prominent.
>>
>> My algorithm - attempts to solve the (PATH, BASE, MEMBER) problem as
>> "dynamically" as possible. PATH and BASE are fairly straight forward -
>> but MEMBER is clearly more complex.
>>
>> PATH: start by looking at the python executable -
> As far as I can tell, any executable can actually link against the Python interpreter.
I am talking about packaging python as "stand-alone"
>> and looking at it's  "blibpath" -
> There also is the loadquery() subroutine in AIX, see https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.basetrf1/loadquery.htm
>
> loadquery(L_GETLIBPATH) "Returns the library path that was used at process exec time.",
> which includes both the environment variable LIBPATH (or LD_LIBRARY_PATH if LIBPATH is unset) and the executable's "blibpath" value.
Not having used loadquery() I do not know if it also, by default, 
returns the default library path that the executable uses for it's own 
(initial) libraries. E.g, when packaging I try to use /opt/lib, rather 
than /usr/lib as I do not want to conflict or overwrite existing 
libraries. So, when I add -L/opt/lib the XCOFF headers of python, the 
executable, include /opt/lib.
>> and using that as the default colon separated list of PATHs
> Question is if we do want to consider _current_ values of environment variable LIBPATH (or LD_LIBRARY_PATH) in addition to the "library path at process exec time"?
In Python2.X, and prior to Python 3.6 - for Linux it was ignored. I 
cannot comment on other platforms. On AIX I know that CDLL("libfoo.so") 
and CDLL("libfoo.a(member.o), RTLD_MEMBER) follow LIBPATH or, if LIBPATH 
is not defined LD_LIBRARY_PATH if it is defined). In short, 
LoadLibrary() may find something different compared to what find_library 
returns.

> to search for BASE.a archive. Once a BASE.a file is found it is examined > for a MEMBER. If all PATH/BASE.a do not find a potential MEMBER then the > PATHs are examined again for PATH/BASE.so. Erm, nope, the AIX linker has a different algorithm (for -lNAME): Iterating over the "library path", the first path entry containing any matching filename (either libNAME.a or libNAME.so) will be used, and no further library path iteration is performed. This one found PATH/filename does have to provide the requested symbol in one way or another. Maybe the linker (ld) follows this. It has been a year since I did my tests for rtld behavior. Above is what I recall. Or maybe I just tested wrong. In any case, I felt it is more 'friendly' to skip over a libfoo.a that does not include a dlopen() suitable object (aka member) and look for a dlopen() suitable .so file. > When a .so file is found that > is returned - versioning must be accomplished via a symbolic link to a > versioned library. I mean, if we are getting to "file" level, it should function as I believe "Linux" is behaving. Whatever the platform thinks is "best".

Linux x066 3.2.0-4-powerpc64 #1 SMP Debian 3.2.78-1 ppc64 GNU/Linux

root@x066:~# find / -name libc.so\* /lib64/libc.so.6 /lib/powerpc-linux-gnu/libc.so.6 /usr/lib64/libc.so /usr/lib/powerpc-linux-gnu/libc.so

root@x066:~# python -m ctypes/util libm.so.6 libc.so.6 libbz2.so.1.0 \<CDLL 'libm.so', handle f7e965e0 at f7c20370> \<CDLL 'libcrypt.so', handle 1096df58 at f7c20370> libcrypt.so.1

michael@x067:~$ uname -a Linux x067 3.16.0-4-powerpc64 #1 SMP Debian 3.16.7-ckt9-3 (2015-04-23) ppc64 GNU/Linux

And, as you look at the details - perhaps 'libm' shows more clearly how python really does not care - much. find_library() may find one thing, but if the less specific exists, then the less specific is loaded.

michael@x067:~$ uname -a Linux x067 3.16.0-4-powerpc64 #1 SMP Debian 3.16.7-ckt9-3 (2015-04-23) ppc64 GNU /Linux michael@x067:~$ uname -a Linux x067 3.16.0-4-powerpc64 #1 SMP Debian 3.16.7-ckt9-3 (2015-04-23) ppc64 GNU/Linux (switching to root, so find does not complain about non-accessible directories) michael@x067:~$ su Password: root@x067:/home/michael# find / -name libc.so\ /usr/lib/powerpc-linux-gnu/libc.so /lib/powerpc-linux-gnu/libc.so.6 root@x067:/home/michael# python -m ctypes/util libm.so.6 libc.so.6 libbz2.so.1.0 \<CDLL 'libm.so', handle f7ea3528 at f7b970b0> \<CDLL 'libcrypt.so', handle 10a74e68 at f7b970b0> libcrypt.so.1 root@x067:/home/michael# find / -name libm.so\ /usr/lib/powerpc-linux-gnu/libm.so /lib/powerpc-linux-gnu/libm.so.6

The linker does not perform such a check, nor does it feel necessary for ctypes.find_library+dlopen as long as it does search similar to the linker. Above shows (find output) - that the names are 'equal' - but where they actually are is 'don't care' from python perspective. That is a low-level detail - platform dependent. And as the documentation says:

Python2.7:

|ctypes.util.||find_library|(/name/)

Try to find a library and return a pathname. /name/ is the library
name without any prefix like /lib/, suffix like |.so|, |.dylib| or
version number (this is the form used for the posix linker option
|-l|). If no library can be found, returns |None|.

The exact functionality is system dependent.

Python3.6.1

|ctypes.util.||find_library|(/name/)

Try to find a library and return a pathname. /name/ is the library
name without any prefix like /lib/, suffix like |.so|, |.dylib| or
version number (this is the form used for the posix linker option
|-l|). If no library can be found, returns |None|.

The exact functionality is system dependent.

On Linux, |find_library()| tries to run external programs (|/sbin/ldconfig|, |gcc|, |objdump| and |ld|) to find the library file. It returns the filename of the library file.

Changed in version 3.6: On Linux, the value of the environment variable |LD_LIBRARY_PATH| is used when searching for libraries, if a library cannot be found by any other means.

Please note: "The exact functionality is system dependent.", so I have not understood how adding _aix.py (_dynloadaix.py if you must have a longer name, the (underscore at the beginning is key (and thinking PEP-20 I thought _aix was a fine name).

But somewhere, perhaps in my 'noobish python attempts that needed upgrading" arose the objection to including anything - passing by the true issue. find_library() is broken.

> The program "dump -H" provides this information for both executables and > archive (aka BASE) members. Eventually we might want to avoid spawning the 'dump' program, but implement reading the XCOFF Object File Format within _ctypes module instead. At least AIX does provide the necessary headers: https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.files/XCOFF.htm On my 'to read' list. > Starting from the "blibpath" values in the executable mean a cpython > packager can add a specific PATH by adding it to > LDFLAGS="-L/my/special/libdir:$LDFLAGS". Note that AIX archives also > have their own "blibpath" - so libraries dynamically loaded may also > follow additional paths that the executable is not aware of (nor need to > be). There is no need for the ctypes module to search libpaths from other Shared Objects than the main executable (and current env vars). OK. But my point is - the environment variable is most generally not defined, and my observation is that the LIBPATHs that rtld searches is: a) the LIBPATH defined in the XCOFF info of the executable when it loads, and b) the LIBPATH specified in the libfoo.a(member) when it needs other libraries. These may not be the same. Although, I must verify (again) part b.

I agree that an executable should not need to care about the LIBPATH needs of libraries it wants to dlopen().

> So - once the PATHS are determined the system is examined looking for > ${PATH}/BASE.a. If a target BASE.a is found, it is examined for a MEMBER > is set to BASE.so (now used a MEMBER.so) . If MEMBER.so is not found > then look for the "highest X[.Y[.Z]] aka MEMBER.so.X[.Y[.Z]] name. If > that is not found check AIX legacy names (mainly shr.o or shr_64.o, > although there are also certain key libraries that have additional > variations (boring)). When ctypes.dlopen is asked to load an Archive Library (either .a or .so) without a specific member, it probably should not immediately dlopen a specific member, but fetch the list of symbols provided by useable members (either Shared Objects without the F_LOADONLY flag, as well as specified in Import Files), and return the handle to some internal symbol->member map instead.

Then, really loading a shared archive member is done by subsequent ctypes.dlsym - where it becomes clear which archive member to load. IMHO (noob me) thinks this is a "new behavior" not described in Python2.7 (or even Python3.6) ctypes documentation - and beyond the scope of what I have been reporting. > Again, if PATH, BASE, MEMBER is not located as a .a archive - look for > libFOO.so in all the PATH directories known to the executable. Nope, see above - please iterate over a libpath list only _once_, and search for each filename while at one path list entry. Actually, I think we largely agree. If I understand your comment correctly you would prefer to search, per directory for libfoo.a (for suitable members), then libfoo.so, then next directory in LIBPATH - rather than my search for libfoo.so only after LIBPATH directories have all 'failed' to find a shareable member.

Our difference is that I put .a archives - at the premium - whereas you put "files" at the premium.

Maybe your way is how rtld (aka InitandLoad() searches), and then that should be the way to go!

However, I'm not sure yet how to identify if we should search for .a or .so first (before .so and .a, respectively): This depends on whether the current executable actually does use runtime linking or not, but I have no idea yet how to figure that out. I have not specifically looked - but I have not seen any application on AIX (in recent years) that only uses static libraries. But probably we may want to use how the Python interpreter (libpython.a or libpython.so) was built. Using "./configure" defaults - looking at my own packaging: $ ./configure --prefix=/opt --sysconfdir=/var/python/etc --sharedstatedir=/var/python/com --localstatedir=/var/python --mandir=/usr/share/man --infodir=/opt/share/info/python --without-computed-gotos

executable: (never released though, am hung on whether to add my non-accepted patch to ctypes or not)

root@x064:[/data/prj/python/python-2.7.13.0]dump -H python

python:

                     \*\*\*Loader Section***
                   Loader Header Information

VERSION# #SYMtableENT #RELOCent LENidSTR 0x00000001 0x000005b2 0x00003545 0x0000006e

IMPfilID OFFidSTR LENstrTBL OFFstrTBL

0x00000005 0x0003080c 0x00006837 0x0003087a

                     \*\*\*Import File Strings***

INDEX PATH BASE MEMBER 0 /usr/vac/lib:/usr/lib:/lib 1 libc.a shr.o 2 libpthreads.a shr_xpg5.o 3 libpthreads.a shr_comm.o 4 libdl.a shr.o

executable Uses rtld for 'other stuff'. FYI: shareable does work (i.e., libpython* is also in the dump -H info), but ../../python (for testing) does not work when "shared". Since most of my time is spent testing "shared" is time-consuming. So, I do not add --shared by default.

root@x064:[/data/prj/python/python-2.7.13.0]ls -l lib* -rw-r----- 1 root 1954 3156377 Jan 13 13:18 libpython2.7.a

No shared members, only static .o files (112 of them)

Uhm, ultimative solution feels complex already, while still some things to decide... I am willing to help. But I can only take 'no' for so long. Then I'll pass the baton.

Again, my apologies for the long delay since your extensive reply. I hope you see I have spent (too) many hours on this. I also hope that you understand I do not see my solution as the only solution (I am not that Dutch ;) !). I just hope for resolution - closer to sooner than to 'never'.

Michael

----------

Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue27435\

On 24/02/2017 09:44, Michael Haubenwallner wrote:

Michael Haubenwallner added the comment:

Let's switch to github-based patches to discuss about: https://github.com/python/cpython/compare/2.7...haubi:bpo-27435/2.7

For the library search path I prefer to use loadquery(L_GETLIBPATH), available via new _ctypes_aix module now, but also used with L_GETINFO in Python/dynload_aix.c already.

Back again - I could not find the _ctypes_aix you mentioned. github was new, and I had no time then. And what I saw in dynload_aix.c I did not understand. I was trying to emulate what I saw in the existing Lib/ctypes (i.e., resolve it in python code) rather than write code in C.

Clearly, you understand that path - I (still) do not. Obviously - making a library call is more efficient than making multiple subprocess calls.

I may be far enough along with git that I might be able to figure out how to "fetch" what you have on github as haubi:bpo-27435/2.7.

This may also be better, i.e., more efficient, ultimately, than what was merged via https://github.com/python/cpython/pull/4507

Now that I have found your work - thank you for all the time you put into this last year!

Michael

----------

Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue27435\

as this is not (likely) to be backported to Python2 (by python, fyi: I do include full ctypes-load_library() support in my "independent" packaging)

and it is "resolved" for Python3-3.7 bpo-26439

this issue may, imho, be closed.

Thanks Michael, finally getting it closed :)

python / cpython

ctypes library loading and AIX - also for 2.7.X (and later) #71622

IMPfilID OFFidSTR LENstrTBL OFFstrTBL