BrokenProcessPool Error when trying to run Multicall

[Zeazy]

Hi everyone,

I am getting the below BrokenProcessPool runtime error when attempting to run the Multi call example script. I am not sure what is causing this and attempts to debug myself lead me down a low-level concurrent futures path that I am not familiar with. Note this error occurs regardless of whether I pass in w3 instance as parameter or set it as an os environment variable.

Can anyone assist here?

Thank you in advance.

Error: concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Example Script:

  w3 = Web3(Web3.HTTPProvider(alchemy_endpoint))
  def from_wei(value):
      return value / 1e18

  multi = Multicall([
      Call(MKR_TOKEN, ['balanceOf(address)(uint256)', MKR_WHALE], [['whale', from_wei]]),
      Call(MKR_TOKEN, ['balanceOf(address)(uint256)', MKR_FISH], [['fish', from_wei]]),
      Call(MKR_TOKEN, 'totalSupply()(uint256)', [['supply', from_wei]]),
  ], _w3 = w3)
  multi()

Full Error Message:

  /Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/bin/python3.10 /Users/hamzaalfadel/Desktop/Cred/Code/test
  _v2/test_file_2.py

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
      exitcode = _main(fd, parent_sentinel)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
      prepare(preparation_data)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
      _fixup_main_from_path(data['init_main_from_path'])
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
      main_content = runpy.run_path(main_path,
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 269, in run_path
      return _run_module_code(code, init_globals, run_name,
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 96, in _run_module_code
      _run_code(code, mod_globals, init_globals,
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/test_file_2.py", line 46, in <module>
      multi()
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/multicall.py", line 57, in __call__
      response = await_awaitable(self.coroutine())
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/utils.py", line 67, in await_awaitable
      return get_event_loop().run_until_complete(awaitable)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
      return future.result()
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/multicall.py", line 62, in coroutine
      batches = await gather([
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/utils.py", line 82, in gather
      raise_if_exception_in(results)
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/utils.py", line 78, in raise_if_exception_in
      raise_if_exception(obj)
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/utils.py", line 74, in raise_if_exception
      raise obj
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/multicall.py", line 97, in fetch_outputs
      _raise_or_proceed(e, len(calls), ConnErr_retries=ConnErr_retries)
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/multicall.py", line 210, in _raise_or_proceed
      raise e
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/multicall.py", line 82, in fetch_outputs
      args = await run_in_subprocess(get_args, calls, self.require_success)
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/utils.py", line 70, in run_in_subprocess
      return await asyncio.get_event_loop().run_in_executor(process_pool_executor, coro, *args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 818, in run_in_executor
      executor.submit(func, *args), loop=self)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/process.py", line 705, in submit
      self._adjust_process_count()
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/process.py", line 683, in _adjust_process_count
      p.start()
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 121, in start
      self._popen = self._Popen(self)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 284, in _Popen
      return Popen(process_obj)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
      super().__init__(process_obj)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
      self._launch(process_obj)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
      prep_data = spawn.get_preparation_data(process_obj._name)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
      _check_not_importing_main()
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
      raise RuntimeError('''
  RuntimeError: 
          An attempt has been made to start a new process before the
          current process has finished its bootstrapping phase.

          This probably means that you are not using fork to start your
          child processes and you have forgotten to use the proper idiom
          in the main module:

              if __name__ == '__main__':
                  freeze_support()
                  ...

          The "freeze_support()" line can be omitted if the program
          is not going to be frozen to produce an executable.
  Traceback (most recent call last):
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/test_file_2.py", line 46, in <module>
      multi()
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/multicall.py", line 57, in __call__
      response = await_awaitable(self.coroutine())
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/utils.py", line 67, in await_awaitable
      return get_event_loop().run_until_complete(awaitable)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
      return future.result()
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/multicall.py", line 62, in coroutine
      batches = await gather([
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/utils.py", line 82, in gather
      raise_if_exception_in(results)
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/utils.py", line 78, in raise_if_exception_in
      raise_if_exception(obj)
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/utils.py", line 74, in raise_if_exception
      raise obj
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/multicall.py", line 97, in fetch_outputs
      _raise_or_proceed(e, len(calls), ConnErr_retries=ConnErr_retries)
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/multicall.py", line 210, in _raise_or_proceed
      raise e
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/multicall.py", line 82, in fetch_outputs
      args = await run_in_subprocess(get_args, calls, self.require_success)
    File "/Users/hamzaalfadel/Desktop/Cred/Code/test_v2/venv/lib/python3.10/site-packages/multicall/utils.py", line 70, in run_in_subprocess
      return await asyncio.get_event_loop().run_in_executor(process_pool_executor, coro, *args, **kwargs)
  concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

[cy-phiz]

same here

[BobTheBuidler]

I've experienced this behavior when a process in the process pool has an exception. #44 will change the default behavior to no-parallelism which should allow you to see the exception causing your issue.

[BobTheBuidler]

This is resolved in 0.6.0