python-trio / purerpc

Native, async Python gRPC client and server implementation supporting asyncio, uvloop, and trio
Apache License 2.0
217 stars 15 forks source link

tests fail on Python 3.8 (macOS) #30

Closed belm0 closed 2 years ago

belm0 commented 3 years ago

e.g.

_____ ERROR at setup of test_grpc_client_large_payload[purerpc_echo_port] ______

cm_gen = <function run_purerpc_service_in_process.<locals>.target_fn at 0x10c175310>

    @contextlib.contextmanager
    def _run_context_manager_generator_in_process(cm_gen):
        parent_conn, child_conn = multiprocessing.Pipe(duplex=False)
        target_fn = _wrap_gen_in_process(child_conn)(cm_gen)

        process = multiprocessing.Process(target=target_fn)
>       process.start()

src/purerpc/test_utils.py:97:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
../../.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
../../.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/context.py:284: in _Popen
    return Popen(process_obj)
../../.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
../../.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
../../.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <Process name='Process-1' parent=73643 initial>
file = <_io.BytesIO object at 0x10c1ddf40>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'run_purerpc_service_in_process.<locals>.target_fn'

../../.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
belm0 commented 3 years ago

It may be specific to macOS.

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Using spawn seems to have various implications for existing code.

https://chrissardegna.com/blog/multiprocessing-changes-python-3-8/#picklingerror-crash

Possible workaround (given that it would only be utilized for tests):

multiprocessing.set_start_method("fork")