fastai / nbdev

Create delightful software with Jupyter Notebooks
https://nbdev.fast.ai/
Apache License 2.0
4.93k stars 488 forks source link

`nbdev_test` breaks on Apple M1 due to `BrokenAccessPool` #1256

Open CarloLepelaars opened 1 year ago

CarloLepelaars commented 1 year ago

nbdev is working great for me on Linux VMs, but every time I try to run nbdev_test on my Macbook Pro M1 I run into the error below. nbdev_export is running fine.

Traceback for nbdev_test call:

Traceback (most recent call last):
  File "/opt/anaconda3/envs/ENV/bin/nbdev_prepare", line 8, in <module>
    sys.exit(prepare())
  File "/opt/anaconda3/envs/ENV/lib/python3.7/site-packages/fastcore/script.py", line 119, in _f
    return tfunc(**merge(args, args_from_prog(func, xtra)))
  File "/opt/anaconda3/envs/ENV/lib/python3.7/site-packages/nbdev/quarto.py", line 238, in prepare
    nbdev.test.nbdev_test.__wrapped__()
  File "/opt/anaconda3/envs/ENV/lib/python3.7/site-packages/nbdev/test.py", line 90, in nbdev_test
    basepath=get_config().config_path, pause=pause, do_print=do_print, **kw)
  File "/opt/anaconda3/envs/ENV/lib/python3.7/site-packages/fastcore/parallel.py", line 117, in parallel
    return L(r)
  File "/opt/anaconda3/envs/ENV/lib/python3.7/site-packages/fastcore/foundation.py", line 98, in __call__
    return super().__call__(x, *args, **kwargs)
  File "/opt/anaconda3/envs/ENV/lib/python3.7/site-packages/fastcore/foundation.py", line 106, in __init__
    items = listify(items, *rest, use_list=use_list, match=match)
  File "/opt/anaconda3/envs/ENV/lib/python3.7/site-packages/fastcore/basics.py", line 66, in listify
    elif is_iter(o): res = list(o)
  File "/opt/anaconda3/envs/ENV/lib/python3.7/concurrent/futures/process.py", line 483, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/opt/anaconda3/envs/ENV/lib/python3.7/concurrent/futures/_base.py", line 598, in result_iterator
    yield fs.pop().result()
  File "/opt/anaconda3/envs/ENV/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/opt/anaconda3/envs/ENV/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

The same error occurs in a Python 3.9 environment. Am I right to think this is an issue specific to Apple M1s hardware or could it be a more general problem? I'm using concurrent.futures.ThreadPoolExecutor functionality in my code so it could be related to that.

hamelsmu commented 1 year ago

I found that I have to export the following env variable on M1 Mac

OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

cc: @jph00

Taytay commented 1 year ago

I bet this is either related or a dupe of: https://github.com/fastai/nbdev/issues/731 https://github.com/fastai/nbdev/issues/693 https://github.com/fastai/nbdev/issues/673