search

vpelletier / python-libusb1

Python ctype-based wrapper around libusb1
GNU Lesser General Public License v2.1
178 stars 67 forks source link

OSError: dlopen failed: library "libusb-1.0.so" not found #78

Open Camcz opened 3 years ago

Camcz commented 3 years ago

Hello, I am building an app using Kivy. When running the app in android using buildozer android run logcat I get the error:

.../.buildozer/android/platform/build-arm64-v8a/build/python-installs/myapp/usb1/libusb1.py", line 157, in _loadLibrary
.../.buildozer/android/platform/build-arm64-v8a/build/other_builds/python3/arm64-v8a__ndk_target_21/python3/Lib/ctypes/__init__.py", line 373, in __init__
09-10 00:28:32.415  9400  9428 I python  :  OSError: dlopen failed: library "libusb-1.0.so" not found
09-10 00:28:32.415  9400  9428 I python  : Python for android ended.

When looking at the libusb1.py file, in the _loadLibrary function I see that my system 'Linux' is not catered for, hence the error (I stand to be corrected).

I thought of creating a recipe so I could edit the libusb1.py file and add the path to the libusb-1.0.so.0 in my computer which is '/lib/x86_64-linux-gnu/libusb-1.so.0' but that is something I do not know since I am new to app development using Kivy and buildozer.

If there are any workarounds for this kind of issue, please help.

vpelletier commented 3 years ago

Hello and thanks for the report.

About _loadLibrary, the general intent is to first try to let the OS linker find the library to load, and only if it fails to apply platform-specific workarounds for common library locations.

So the ideal fix would be to somehow get libusb-1.0.so to become reachable by the linker. I have no experience of this level with android. Is it present in android at all ? Otherwise, I guess this means each application needs to bundle its own copy of it. Also, I note that the logcat output mentions arm64: you will need to bundle a libusb built for the correct architecture, x86_64 would likely fail with exactly the same error. Then you will need to figure out where the linker will be able to find the library.

On this last point, I have just pushed a proof-of-concept change to allow programs to give usb1.USBContext a library to use, allowing the program to get full control of the library lookup logic. It is not ready for release, but ideas and feedback is very welcome. More discussion about this happen in #37 . The code is in the wip branch.

vpelletier commented 3 years ago

Checking libusb usage on android, it looks like there are a few more roadblocks:

For the first point, the difficulty is going to be the lifecycle reversal: normally, USBDevice outlives USBDeviceHandle, but not in this case: libusb_close (closing the handle) also destroys the device.

mcuee commented 3 years ago

Take note libusb itself have multiple issues under Android (non rooted device).

Possible now: https://github.com/libusb/libusb/pull/830 (which is inline with the above comments) Next step: https://github.com/libusb/libusb/pull/874

I tried to get pyusb to work under Termux Android but failed: https://github.com/pyusb/pyusb/issues/285

vpelletier commented 3 years ago

Next step: libusb/libusb#874

This looks very good. It looks like it would solve both of the above roadblocks.

git-developer commented 1 month ago

I receive this error

OSError: dlopen failed: library "libusb-1.0.so" not found

on Linux as user of a python application that embeds python-libusb1. It seems that python-libusb1 tries to load from a file libusb-1.0.so that does not exist on my system, while libusb-1.0.so.0 does.

A symlink fixes the problem.

I had a look at the packages of current Debian and Ubuntu distros and found out that

This explains the behavior. What is the proper fix? Is the problem caused by a bug in python-libusb1 or do I need to tweak my setup?

vpelletier commented 1 month ago

Note that this bug is about android, which has its own set of peculiarities about native library support. But it's not like there is a lot of traffic here anyway.

IMHO this is a python-libusb1 bug.

I very recently discovered by chance such apparent shift in the .so symlink location on current Debian sid. I have not looked into the reason nor confirmed this is a recent change (I have the libusb -dev package installed on most of my machines anyway), but I think explicitly depending on the major ABI version makes some sense: if the libusb project ever makes an incompatible evolution its ABI (so I would expect it would then be distributed as .so.1) I would rather python-libusb1 fail to locate the .so.0 than accidentally link against the wrong ABI. I start looking into this.

vpelletier commented 1 month ago

A first cursory check of Debian packager documentation suggests this is not a new change. The closest entry I can find in the changelog dates from 2008, but it is not even clear whether this is about .so vs. .so.0, and I do not immediately see a link to such version to check.

Then, checking my code some more, I realize that it may have used the fallback codepath more than anticipated: if loading libusb-1.0.{dll,so,dylib} fails (which is indeed the case when libusb-1.0.so does not exist) it calls ctypes.util.find_library with usb-1.0 (or usb on FreeBSD) which - at least on my system - does find libusb-1.0.so.0. Then, it re-raises the original error (with the original filename, so .so) if find_library fails to locate that library.

But python's ctype documentation of find_library says that this relies on external commands:

On Linux, find_library() tries to run external programs (/sbin/ldconfig, gcc, objdump and ld) to find the library file. It returns the filename of the library file.

so the outcome may differ when any is unavailable... Also:

If wrapping a shared library with ctypes, it may be better to determine the shared library name at development time, and hardcode that into the wrapper module instead of using find_library() to locate the library at runtime.

Indeed, strace'ing python3 on my machine shows it forks a /sbin/ldconfig -p and finds libusb-1.0.so.0 in its output. Could you check ?

$ strace -f python3 -c "import ctypes.util; ctypes.util.find_library('usb-1.0')"
[...]
pipe2([6, 7], O_CLOEXEC)                = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
vfork(strace: Process 1485713 attached
 <unfinished ...>
[pid 1485713] close(4)                  = 0
[pid 1485713] close(6)                  = 0
[...]
[pid 1485713] execve("/sbin/ldconfig", ["/sbin/ldconfig", "-p"], 0x7f5108b1b490 /* 2 vars */ <unfinished ...>
[...]
git-developer commented 1 month ago

Thanks for your quick response!

I ran the strace command above and can confirm that /sbin/ldconfig (and later on usr/bin/ld) is called in my environment.

Still the library is not found without a symlink:

$ find . -name libusb-1.0*
./lib/x86_64-linux-gnu/libusb-1.0.so.0.2.0
./lib/x86_64-linux-gnu/libusb-1.0.so.0
$ grep EXEC AppRun.env
APPDIR_EXEC_PATH=$APPDIR/usr/bin/python3
APPDIR_EXEC_ARGS=-c "import ctypes.util; print(ctypes.util.find_library('usb-1.0'))"
$ env -C runtime/compat ../../AppRun
None
$ ln -sr lib/x86_64-linux-gnu/libusb-1.0.so.0 lib/x86_64-linux-gnu/libusb-1.0.so
$ env -C runtime/compat ../../AppRun
libusb-1.0.so.0

(The application is wrapped in an AppImage. In an AppImage, the command AppRun prepares the environment and then runs the main binary given by the APPDIR_EXEC_ variables.)

git-developer commented 1 month ago

The field of loading shared libraries is new to me. I just read a helpful blog post about it, and I get the impression that a library file name without a trailing ABI number is meant for development purposes only.

From ctypes - Loading dynamic link libraries:

On Linux, it is required to specify the filename including the extension to load a library

... and the example contains a file name including ABI version.

When running ldconfig in my environment, the only symlink printed / created is libusb-1.0.so.0 -> libusb-1.0.so.0.2.0.

vpelletier commented 1 month ago

I pushed a pair of changes to master which should address this issue. I am unfortunately not ready to release the next version just yet (there is still work to do in the next branch).

git-developer commented 1 month ago

Thanks! I have a workaround, so there's no urgent need for a release.