multiprocessing.joinablequeue error when running demo code in jupyter lab

[drhades]

Jupyter Lab: 1.2.6 Python: 3.8.0

Running the following demo code:

Background Jupyter service error logging (and the above cell sinked in running status, cannot be stopped):

Process Worker-2: Process Worker-1: Process Worker-3: Traceback (most recent call last): Traceback (most recent call last): File "~/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap self.run() File "~/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 235, in run self._run_tasks() File "~/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap self.run() File "~/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 250, in _run_tasks message = self.task_queue.get() File "~/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/queues.py", line 116, in get return _ForkingPickler.loads(res) File "~/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 235, in run self._run_tasks() File "~/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 250, in _run_tasks message = self.task_queue.get() File "~/.pyenv/versions/3.8.0AttributeError: Can't get attribute 'task_loop' on <module 'main' (built-in)> Traceback (most recent call last): /lib/python3.8/multiprocessing/queues.py", line 116, in get return _ForkingPickler.loads(res) File "~/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap self.run() File "~/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 235, in run self._run_tasks() File "~/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 250, in _run_tasks message = self.task_queue.get() AttributeError: Can't get attribute 'task_loop' on <module 'main' (built-in)> File "~/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/queues.py", line 116, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'task_loop' on <module 'main' (built-in)>

[TaiSakuma]

Thank you for letting me know. I'll have a look.

[drhades]

For reference, defining task_loop() in another python file and importing, it'll be ok.

[TaiSakuma]

Yes, I'm just working on this now. The code in this branch might work.

[TaiSakuma]

I released the version 0.9.8.

It works fine in Python 3.8 on Jupyter Notebook in my environment.

Please let me know if the problem persists.

The cause of the problem appears to be the change in the default start method in multiprocessing in Python 3.8. The default start method was "fork" until Python 3.7. But it changed to "spawn" in Python 3.8. The spawn method has more restrictions on how the main module is written as described in this section of the document for mulriprocessing. The mantichora version 0.9.8 uses the "fork" mode.

[drhades]

@TaiSakuma Thanks for your quick fix and detail explaination!

After upgrading to 0.9.8, the above demo code work well, but I get the following error:

when using mantichora background, code like:

def task_func(*args)
    for i in atpbar(...):
        do some network things
    return something

def parallel_func()
    split something into K parts
    task_partial = partial(task_func, ...)
    with mantichora() as mcore:
        for i in range(K):
            mcore.run(task_partial, ...)
        mcore.receive_all()

It works fine by 0.9.7.

[TaiSakuma]

Thank you for letting me know. I will look into it.

[TaiSakuma]

I released mantichora version 0.9.9.

I added the option mp_start_method to mantichora() so that uses can choose the start method of multiprocessing. The possible values are fork, spawn, and forkserver. The default is fork. For example, if you would like to use the spawn method, which was used in mantichora v0.9.7 in Python 3.8, you can do so as

mantichora(mp_start_method='spawn')

The spawn and forkserver mothods have more restrictions, which are already mentioned above in this issue.

If you need to use the fork method, you might get the same error of may have been in progress in another thread when fork() was called. If you are getting the same error and you are on MacOS, setting the environment variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES might solve the problem as suggested at StackOverflow here, which you might have found already.

Please let me know if this doesn't solve the problem or causes other problems.

Thank you.

[TaiSakuma]

I assume that the issue has been resolved. Please create a new issue if not.