Fix multiprocessing with `spawn` & top level imports

[bryango]

This fixes test_fibonacci_multiprocessing on MacOS (darwin), which was failing with:

_pickle.PicklingError: Can't pickle <function rebuild_connection at ...>: import of module 'multiprocessing.connection' failed

This is due to the fact that multiprocessing on darwin defaults with the start method of spawn, instead of fork in other POSIX environments.

• The only start method supported by all platforms is 'spawn'; moreover, the default method will move away from 'fork' in python 3.14. We thus set method='spawn' for consistency across platforms.

• fibonacci_multiprocessing makes use of the multiprocessing helper stem.util.system.DaemonTask, whose arguments must be imported at the top level for it to be properly pickled and spawned.

See: https://docs.python.org/3/library/multiprocessing.html

[atagar]

Thanks. This looks great to me! Couple questions...

+ORIGINAL_PATH = list(sys.path)
+sys.path.append(EXAMPLE_DIR)

We previously toggled sys.path in the setUp and tearDown methods so global state only changed while our tests run.

Unless I'm missing something we should do sys.path = ORIGINAL_PATH after the following import, and revert setUp to not reference ORIGINAL_PATH.

+from multiprocessing import connection  # noqa: E402, F401
+if stem.util.system.is_windows():
+  from multiprocessing import popen_spawn_win32  # noqa: F401
+else:
+  from multiprocessing import popen_spawn_posix  # noqa: F401

What is the purpose of this part? I'm not spotting the reason for these popen imports.

Thanks!

[bryango]

We previously toggled sys.path in the setUp and tearDown methods so global state only changed while our tests run.

Unless I'm missing something we should do sys.path = ORIGINAL_PATH after the following import, and revert setUp to not reference ORIGINAL_PATH.

Yes, you are right! Sorry I missed this... Force-pushed and fixed.

+from multiprocessing import connection  # noqa: E402, F401
+if stem.util.system.is_windows():
+  from multiprocessing import popen_spawn_win32  # noqa: F401
+else:
+  from multiprocessing import popen_spawn_posix  # noqa: F401

This is to prevent PicklingErrors of import of module 'multiprocessing.{connection,popen_spawn_win32,popen_spawn_posix}' failed. I am not sure if this is the right way to do it though.

Note that if docs/_static/example/fibonacci_multiprocessing.py is directly called with python, these imports are not necessary. The issue seems to be that we are calling it from the test module, with fibonacci_multiprocessing.main(), such that the spawned process fails to locate the required modules multiprocessing.{connection,popen_spawn_win32,popen_spawn_posix} automatically.

[bryango]

Note that if docs/_static/example/fibonacci_multiprocessing.py is directly called with python, these imports are not necessary.

I have pushed a new commit which directly calls fibonacci_multiprocessing.py with subprocess.run. This should be a much cleaner fix.

[atagar]

Looks great to me! Thanks, pushed.