search

chaquo / chaquopy

Chaquopy: the Python SDK for Android
https://chaquo.com/chaquopy/
MIT License
842 stars 133 forks source link

Support standard library `multiprocessing` module #963

Open mhsmith opened 1 year ago

mhsmith commented 1 year ago

Originally posted by @bin-san at https://github.com/beeware/beeware/issues/262

In the doc of chaquopy it says:

multiprocessing Because Android doesn’t support POSIX semaphores, most of the multiprocessing APIs will fail with the error “This platform lacks a functioning sem_open implementation”. The simplest solution is to use multiprocessing.dummy instead.

And multiprocess.dummy is nothing but a fancy name of threading.

multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module.

So, to use "real" multiprocessing, we should give our platform(android) what it lacks(library that gives a sem_open implementation). I have came across Termux which is a beautiful terminal emulator with a linux environment. In termux, I can install python 3.11 and use the multiprocessing module. So, to find what library is causing multiprocessing to work, I copied the python interpreter and its standard library from Termux's internal storage to /data/local/tmp to create a seperate environment for python. Then I ran python from the /data/local/tmp, as expected it did not run. The cause is libandroid-posix-semaphore.so could not be found.

Voila!!! we got the library name that can give our python app the strength of multiprocessing. I soon copied the libandroid-posix-semaphore.so from termux's internal storage and paste it into /data/local/tmp. Then When I ran, everything works like a charm.

So, multiprocessing is achievable on android. Now, I have no idea, how to use this libandroid-posix-semaphore.so in my app. What I have to do, is put the library somewhere in my app and set the LD_LIBRARY_PATH to the directory. Can anyone help me how to achieve that?

mhsmith commented 1 year ago

The test for the sem_open function is done at compile time, so simply providing the library at runtime won't be enough – Python would have to be rebuilt to use it. I don't have time to look into this just now, but if you're interested, the Python build scripts can be found in the target directory of this repository.

mhsmith commented 1 year ago

Here are some links I found on this subject a few years ago:

I don't know how libandroid-posix-semaphore works, but it may be similar to this idea:

We do IPC between processes using Unix domain sockets, with abstract socket addresses; we fork off a tiny server that listens on the socket and forwards messages between the processes.

What if a single server process (not necessarily the forkserver) also holds all of the inter-process locks? In fact, there's a multiprocessing.Manager class which already implements exactly that.