Couldn't find `libc`'s `posix_memalign` to allocate memory in MacOS Big Sur

[speglich]

In MacOS Big Sur dylibs are removed from the disk when they are added to the shared dyld cache. Thus, ctypes.utils.find_libary cannot find the libraries, as we were used to.

Seems that this issue was solved here for cpython 3.9.1 and any patch is necessary. However it is only available in the release candidate of the Apple Universal Binary version.

Applications that must be compatible with older python versions are writing patches, like this:

elif os.name == "posix" and sys.platform == "darwin":
    from ctypes.macholib.dyld import dyld_find as _dyld_find
    def find_library(name):
        possible = ['@executable_path/../lib/lib%s.dylib' % name,
                    'lib%s.dylib' % name,
                    '%s.dylib' % name,
                    '%s.framework/%s' % (name, name)]
        for name in possible:
            try:
                return _dyld_find(name)
            except ValueError:
                continue
        return None

Following the @FabioLuporini suggestion we could implement the same patch like this:

try:
    ... = find_library(...)
except ...:
    # Special cases
    if os.name == "posix" and sys.platform == "darwin":
         ...
    else:
         <suitable error message>

[mloubout]

Catalina was messed up and BigSur was expected to be worse. It's also just released and half of it is obviously buggy. I don't know if we wanna dog that deep into fixing Apple fuck ups but that's just me. (I would also recommend against upgradong to new IOS until known they didn't break code again and libraries for the nth time)

[FabioLuporini]

But if some new student with a new mac arrives, what do we tell him? pls downgrade the OS? after all, we claim OSX support. I agree with Mathias it's a PITA, but as long as it's a small non-invasive patch, I'm happy to bring it in

[speglich]

I'm working on this patch right now, which so far seemed to be a simple thing. From what I read in the developer documentation:

Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

[speglich]

If we use static paths on ctypes.CDLL(<static_path>), when os.name == "posix" and sys.platform == "darwin" and MacOS >= 10.16 is a bad idea?

[speglich]

This patch solved the problem:

    @classmethod
    def initialize(cls):

        handle = find_library('c')

        if handle == None and os.name == "posix" and sys.platform == "darwin":
            handle = '/usr/lib/libc.dylib'

        if handle is not None:
            cls.lib = ctypes.CDLL(handle)

[FabioLuporini]

• if handle is None and ... (not ==)
• probably you wanna us os.exists(...) to make sure libc.dylib actually exists? or is it unnecessary?

anyway, looking good

[speglich]

If I understand well, any code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail, so so.path.exists will fail (I tried and failed hehe).

[FabioLuporini]

and how will ctypes.CDLL(handle) behave if '/usr/lib/libc.dylib' doesn't exist?

[speglich]

Nice, ctypes.CDLL(handle) will fail trowing an message:

 File "/opt/miniconda3/envs/devito/lib/python3.9/ctypes/__init__.py", line 382, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(<path>): image not found

I could treat this exception (with a better explanation) after ctypes.CDLL(handle), like this:

    def initialize(cls):
        handle = find_library('c')

        if handle is None and os.name == "posix" and sys.platform == "darwin":
            handle = '/usr/lib/libc.dylib'

        if handle is not None:
            try:
                cls.lib = ctypes.CDLL(handle)
            except OSError:
                raise RuntimeError("Couldn't find `libc`'s `posix_memalign` to "
                                "allocate memory")

When handle is None CDLL returns None to cls.lib, maybe set cls.lib as None could be a option too (I don't know if this is good practice), once this initialize method is treat with the same RuntimeError.