loky crash when used from a uwsgi process

[ogrisel]

When issuing a HTTP request on a uswgi app that uses loky one get the following error messages:

unable to load configuration from from loky.backend.semaphore_tracker import main; main(10)
Fri Apr 19 16:39:13 2019 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request / (ip 127.0.0.1) !!!
/home/ogrisel/miniconda3/envs/pip/lib/python3.7/site-packages/loky/backend/semaphore_tracker.py:74: UserWarning: semaphore_tracker: process died unexpectedly, relaunching.  Some semaphores might leak.
  warnings.warn('semaphore_tracker: process died unexpectedly, '
/home/ogrisel/miniconda3/envs/pip/bin/uwsgi: unrecognized option '--process-name'
getopt_long() error
/home/ogrisel/miniconda3/envs/pip/bin/uwsgi: unrecognized option '--process-name'
getopt_long() error
/home/ogrisel/miniconda3/envs/pip/bin/uwsgi: unrecognized option '--process-name'
getopt_long() error
unable to load configuration from from loky.backend.semaphore_tracker import main; main(11)
/home/ogrisel/miniconda3/envs/pip/bin/uwsgi: unrecognized option '--process-name'
getopt_long() error

and then:

Traceback (most recent call last):
/home/ogrisel/miniconda3/envs/pip/bin/uwsgi: unrecognized option '--process-name'
getopt_long() error
  File "loky_wsgi_app.py", line 8, in application
    result = e.submit(lambda: 42).result()
  File "/home/ogrisel/miniconda3/envs/pip/lib/python3.7/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/home/ogrisel/miniconda3/envs/pip/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker. The exit codes of the workers are {EXIT(1), EXIT(1), EXIT(1)}

Steps to reproduce

• Install uswgi: for some reason I could not compile it with recent gcc versions, but it works with clang, e.g.:

CC=clang-7 pip install uswgi

• write a small uswgi app that uses loky, for instance a file named "loky_wsgi_app.py" with the following content:

from loky import get_reusable_executor

def application(env, start_response):
    e = get_reusable_executor()
    result = e.submit(lambda: 42).result()

    start_response('200 OK', [('Content-Type','text/html')])
    return [b"%d" % result]

Start the server:

uwsgi --http :9999 --wsgi-file loky_wsgi_app.py

and navigate to http://localhost:9999 .

[ogrisel]

For the sake of completeness, I also tried with gunicorn and there is no problem:

$ gunicorn loky_wsgi_app:application
[2019-04-19 16:55:05 +0200] [17233] [INFO] Starting gunicorn 19.9.0
[2019-04-19 16:55:05 +0200] [17233] [INFO] Listening at: http://127.0.0.1:8000 (17233)
[2019-04-19 16:55:05 +0200] [17233] [INFO] Using worker: sync
[2019-04-19 16:55:05 +0200] [17236] [INFO] Booting worker with pid: 17236

[ogrisel]

I believe the difference is that uwsgi is probably a C binary that embeds a Python interpreter, therefore it's not possible to use sys.executable to get a Python interpreter from the command line.

[ogrisel]

To fix this we basically need to find a python executable in the parent folder of the uwsgi executable pointed by sys.executable that can import loky from the same installation path (under that is under site-packages folder).

If several Python versions are installed this might be a bit expensive to do so this should be cached.

[seanjkanderson]

Are there any known work-arounds? I am experiencing this issue as well.

[seanjkanderson]

The work-around I was able to find: https://stackoverflow.com/questions/54751297/uwsgi-and-joblib-semaphore-joblib-will-operate-in-serial-mode

[ogrisel]

If someone wants to volunteer to implement the strategy described here: https://github.com/joblib/loky/issues/207#issuecomment-485137208