search

spatialaudio / jackclient-python

🂻 JACK Audio Connection Kit (JACK) Client for Python :snake:
https://jackclient-python.readthedocs.io/
MIT License
131 stars 26 forks source link

Library not found on Apple silicon #120

Open HaHeho opened 1 year ago

HaHeho commented 1 year ago

This concerns an Apple M1 system. I'm surprised importing the module fails entirely and has not been detected yet. Maybe nobody has tried it yet? I assume this may be a more general issue and not exclusive to my device since it is relatively vanilla.

I have no idea how to debug this and find where it actually looks for the library and/or how to modify the locations. Do you have any idea?

mgeier commented 1 year ago

I don't know any expected platform-specific behavior, but it seems that ctypes.util.find_library('jack') returns None instead of the name/path of the library.

This is part of the standard library, so this part is independent of the jack module up to this point.

Here is some discussion about find_library() on macOS: https://github.com/spatialaudio/python-sounddevice/issues/130#issuecomment-379237216

Can you try find_library('libjack.dylib') (or whatever the library file is actually called)?

It might also be interesting to try different Python versions.

HaHeho commented 1 year ago

Thanks for the response. I checked what libraries are available in different places on the M1 system, with the following results:

$ ls -g /opt/homebrew/lib/jack/
total 2824
-r--r--r--@ 1 admin  158768 Feb 18 16:18 audioadapter.so
-r--r--r--@ 1 admin  204032 Feb 18 16:18 jack_coreaudio.so
-r--r--r--@ 1 admin  230256 Feb 18 16:18 jack_coremidi.so
-r--r--r--@ 1 admin   70256 Feb 18 16:18 jack_dummy.so
-r--r--r--@ 1 admin   70032 Feb 18 16:18 jack_loopback.so
-r--r--r--@ 1 admin  130976 Feb 18 16:18 jack_net.so
-r--r--r--@ 1 admin  150240 Feb 18 16:18 jack_netone.so
-r--r--r--@ 1 admin   92816 Feb 18 16:18 jack_proxy.so
-r--r--r--@ 1 admin   85984 Feb 18 16:18 netadapter.so
-r--r--r--@ 1 admin  174304 Feb 18 16:18 netmanager.so
-r--r--r--@ 1 admin   59168 Feb 18 16:18 profiler.so

$ ls -g /opt/homebrew/lib/jack
lrwxr-xr-x@ 1 admin  30 May 20 17:21 /opt/homebrew/lib/jack -> ../Cellar/jack/1.9.22/lib/jack

ls -g /opt/homebrew/Cellar/jack/1.9.22/lib/
total 3656
drwxr-xr-x@ 13 admin      416 Feb 18 16:18 jack
lrwxr-xr-x@  1 admin       15 Feb  2 12:04 libjack.0.1.0.dylib -> libjack.0.dylib
-rw-r--r--@  1 admin   531184 Feb 18 16:18 libjack.0.dylib
lrwxr-xr-x@  1 admin       15 Feb  2 12:04 libjack.dylib -> libjack.0.dylib
lrwxr-xr-x@  1 admin       18 Feb  2 12:04 libjacknet.0.1.0.dylib -> libjacknet.0.dylib
-r--r--r--@  1 admin   202944 Feb 18 16:18 libjacknet.0.dylib
lrwxr-xr-x@  1 admin       18 Feb  2 12:04 libjacknet.dylib -> libjacknet.0.dylib
lrwxr-xr-x@  1 admin       21 Feb  2 12:04 libjackserver.0.1.0.dylib -> libjackserver.0.dylib
-rw-r--r--@  1 admin  1133792 Feb 18 16:18 libjackserver.0.dylib
lrwxr-xr-x@  1 admin       21 Feb  2 12:04 libjackserver.dylib -> libjackserver.0.dylib
drwxr-xr-x@  3 admin       96 Feb 18 16:18 pkgconfig

$ python
Python 3.9.16 (main, Mar  8 2023, 04:29:24)
[Clang 14.0.6 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes.util
>>> print(ctypes.util.find_library('jack'))
None
>>> print(ctypes.util.find_library('libjack.dylib'))
None

Python does not recognize either. Do you have a recommendation on how to modify the system configuration so that the libraries are recognized? Then we could think about how to solve the issue or whether it should be propagated to the JACK community, or how the homebrew formula is set up on Apple Silicon.

FYI, on my Intel Mac, things are in slightly different places, most notably that libjack.dylib is directly in /usr/local/lib/ linking to the actual location.

$ ls -g /usr/local/lib/jack/
total 2648
-r--r--r--  1 admin  139392 Feb 15 14:20 audioadapter.so
-r--r--r--  1 admin  184336 Feb 15 14:20 jack_coreaudio.so
-r--r--r--  1 admin  210416 Feb 15 14:20 jack_coremidi.so
-r--r--r--  1 admin   51768 Feb 15 14:20 jack_dummy.so
-rwxr-xr-x  1 admin   34344 Apr  3 19:33 jack_inprocess.dylib
-rwxr-xr-x  1 admin   35800 Apr  3 19:33 jack_internal_metro.dylib
-rwxr-xr-x  1 admin   50848 Apr  3 19:33 jack_intime.dylib
-r--r--r--  1 admin   51344 Feb 15 14:20 jack_loopback.so
-r--r--r--  1 admin  112200 Feb 15 14:20 jack_net.so
-r--r--r--  1 admin  131608 Feb 15 14:20 jack_netone.so
-r--r--r--  1 admin   74088 Feb 15 14:20 jack_proxy.so
-r--r--r--  1 admin   67456 Feb 15 14:20 netadapter.so
-r--r--r--  1 admin  138728 Feb 15 14:20 netmanager.so
-r--r--r--  1 admin   40576 Feb 15 14:20 profiler.so

$ ls -g /usr/local/lib/jack
lrwxr-xr-x  1 admin  30 Feb 15 14:20 /usr/local/lib/jack -> ../Cellar/jack/1.9.22/lib/jack

$ ls -g /usr/local/Cellar/jack/1.9.22/lib/jack
total 2648
-r--r--r--  1 admin  139392 Feb 15 14:20 audioadapter.so
-r--r--r--  1 admin  184336 Feb 15 14:20 jack_coreaudio.so
-r--r--r--  1 admin  210416 Feb 15 14:20 jack_coremidi.so
-r--r--r--  1 admin   51768 Feb 15 14:20 jack_dummy.so
-rwxr-xr-x  1 admin   34344 Apr  3 19:33 jack_inprocess.dylib
-rwxr-xr-x  1 admin   35800 Apr  3 19:33 jack_internal_metro.dylib
-rwxr-xr-x  1 admin   50848 Apr  3 19:33 jack_intime.dylib
-r--r--r--  1 admin   51344 Feb 15 14:20 jack_loopback.so
-r--r--r--  1 admin  112200 Feb 15 14:20 jack_net.so
-r--r--r--  1 admin  131608 Feb 15 14:20 jack_netone.so
-r--r--r--  1 admin   74088 Feb 15 14:20 jack_proxy.so
-r--r--r--  1 admin   67456 Feb 15 14:20 netadapter.so
-r--r--r--  1 admin  138728 Feb 15 14:20 netmanager.so
-r--r--r--  1 admin   40576 Feb 15 14:20 profiler.so

$ python
Python 3.8.16 (default, Mar  1 2023, 21:19:10)
[Clang 14.0.6 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes.util
>>> print(ctypes.util.find_library('jack'))
/usr/local/lib/libjack.dylib
>>> print(ctypes.util.find_library('libjack.dylib'))
/usr/local/lib/libjack.dylib

$ ls -g /usr/local/lib/libjack.dylib
lrwxr-xr-x  1 admin  39 Feb 15 14:20 /usr/local/lib/libjack.dylib -> ../Cellar/jack/1.9.22/lib/libjack.dylib
mgeier commented 1 year ago

Thanks for the information!

Now I remember that: https://github.com/bastibe/python-soundfile/pull/311

The problem seems to be that the libraries installed by brew on M1 are installed into /opt/homebrew/lib/, while the Intel ones are in /usr/local/lib/.

I would consider this a bug, but I'm not sure if this has to be solved by Python or by brew.

The soundfile module has implemented a (kinda dirty) work-around in https://github.com/bastibe/python-soundfile/pull/311. In the meantime, this has been changed again and it now looks like this: https://github.com/bastibe/python-soundfile/blob/0a8071d1b04c8e45b2465d05866994ec2df8b1f8/soundfile.py#L184-L192

I would hope that this will be fixed by Python or brew, but who knows if that'll happen.

Maybe we also have to implement a work-around?

HaHeho commented 1 year ago

The problem seems to be that the libraries installed by brew on M1 are installed into /opt/homebrew/lib/, while the Intel ones are in /usr/local/lib/.

Right. I validated that on M1 creating a symbolic link with sudo ln -s /opt/homebrew/lib/libjack.dylib /usr/local/lib/libjack.dylib makes the library available, and the package can be imported.

I would consider this a bug, but I'm not sure if this has to be solved by Python or by brew.

The soundfile module has implemented a (kinda dirty) work-around in bastibe/python-soundfile#311. In the meantime, this has been changed again and it now looks like this: https://github.com/bastibe/python-soundfile/blob/0a8071d1b04c8e45b2465d05866994ec2df8b1f8/soundfile.py#L184-L192

I would hope that this will be fixed by Python or brew, but who knows if that'll happen.

Maybe we also have to implement a work-around?

I will look later and try to understand how soundfile did it. This seems to be a bigger issue in general. There isn't a straightforward way of how to add /opt/homebrew/lib/ to the standard search path for where libraries are looked for?

mgeier commented 1 year ago

There isn't a straightforward way of how to add /opt/homebrew/lib/ to the standard search path for where libraries are looked for?

I think this should already be the case, and because it's not, I guess it is a bug.

Other than that, there are some environment variables like LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, DYLD_FALLBACK_LIBRARY_PATH, LD_PRELOAD and maybe more.

On Linux, there is /etc/ld.so.conf, but I don't know if something similar exists on macOS.

HaHeho commented 1 year ago

The following methods make the library available to python so that the

  1. Creating a symbolic link of the JACK library from brew lib to usr lib: ln -s /opt/homebrew/lib/libjack.dylib /usr/local/lib/libjack.dylib

    1. Adding the brew lib path to DYLD_LIBRARY_PATH (which did not exist on my system before): export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/opt/homebrew/lib

    Both of these options require modifications of the local system configuration by the user. Adding (2) to your shell startup config may be helpful for the availability of homebrew libraries on M1 in general. However, from what I remember reading somewhere regarding this general discussion around the destination change of homebrew on M1, manipulating DYLD_LIBRARY_PATH is also something developers and experienced users may want to do with care.

    Therefore, I assume implementing a (temporary) fallback solution inside jack-client, similar to how it is done in python-soundfile (see here), seems like the best solution?

mgeier commented 1 year ago

Yeah, I think the fallback solution would be the most pragmatic for now.