search

CleanCut / green

Green is a clean, colorful, fast python test runner.
MIT License
785 stars 75 forks source link

Multiproccessing freeze on python 3.8.6 and macOS Big Sur #242

Closed sodul closed 3 years ago

sodul commented 3 years ago

First thank you for this simple and easy to use tool to run python unittests.

OS: macOS 11.1 Big Sur pyenv: 1.2.21 python: 3.8.6 green: 3.2.5 coverage: 5.3

When we run some of our test targets with multiprocessing enabled green will freeze in the middle of the tests.

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 513, in _handle_workers
    cls._maintain_pool(ctx, Process, processes, pool, inqueue,
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 337, in _maintain_pool
    Pool._repopulate_pool_static(ctx, Process, processes, pool,
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 319, in _repopulate_pool_static
    w = Process(ctx, target=worker,
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 82, in __init__
    assert group is None, 'group argument must be None for now'
AssertionError: group argument must be None for now
......................

If we CTRL+C we get a larger stack trace, one for each thread, here is a relevant snippet:

Process DaemonlessPoolWorker-10:
Process DaemonlessPoolWorker-3:
Process DaemonlessPoolWorker-1:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/site-packages/green/process.py", line 197, in worker
    task = get()
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/queues.py", line 355, in get
    with self._rlock:
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/site-packages/green/process.py", line 197, in worker
    task = get()
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/queues.py", line 355, in get
    with self._rlock:
  File "/Users/stephane/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt

A workaround is for us to run the tests sequentially, but would like to avoid that as the same version of the packages and tests run completely fine in macOS 10.15 Catalina and in our Ubuntu based containers.

We are not sure if the issue is with green specifically or if python 3.8.6 has a compatibility issue with Big Sur.

sodul commented 3 years ago

I tried to reproduce with pytest and give it 12 thread but there was no issue even in coverage mode.

The coverage and junit (enabled/disabled) do not change anything, setting -s 1 works.

Unfortunately this is on private code and I cannot share the specific tests that cause the issue. These code under test do not use multiprocessing or multithreading, but we do use the newer asyncio feature of Python 3.

CleanCut commented 3 years ago

First thank you for this simple and easy to use tool to run python unittests.

You are welcome!

A workaround is for us to run the tests sequentially, but would like to avoid that as the same version of the packages and tests run completely fine in macOS 10.15 Catalina and in our Ubuntu based containers.

We are not sure if the issue is with green specifically or if python 3.8.6 has a compatibility issue with Big Sur.

Unfortunately this is on private code and I cannot share the specific tests that cause the issue. These code under test do not use multiprocessing or multithreading, but we do use the newer asyncio feature of Python 3.

I'm afraid there is nothing I can do at the moment, as I don't have access to Big Sur yet, nor a code example to reproduce the problem if I did. I will get Big Sur eventally -- in the meantime could you work on providing an sharable example test that triggers the problem?

sodul commented 3 years ago

I think the right approach for now is to wait for pyenv to be updated and try again with the next release. We have some other test suites that just crash right away with a segmentation fault when running through green. So this really points to an issue with the python interpreter.

Everything works fine with the same python versions and requirements.txt on Catalina and under Docker.

I mostly wanted to report so that others can know about the issue before they decide to upgrade, or if they are forced to upgrade since new MacBooks are shipping with Big Sur.

BTW, this is on Intel. I do not have access to M1 machines yet.

sodul commented 3 years ago

Python 3.8.7 was released today with a big warning that 3.8 is not compatible with Big Sur yet. Considering Big Sur ships with python 3.8.2 built-in that is a bit surprising, but explains a few things.

macOS 11 Big Sur not fully supported Python 3.8.7 is not yet fully supported on macOS 11 Big Sur. It will install on macOS 11 Big Sur and will run on Apple Silicon Macs using Rosetta 2 translation. However, a few features do not work correctly, most noticeably those involving searching for system libraries (vs user libraries) such as ctypes.util.find_library() and in Distutils. This limitation affects both Apple Silicon and Intel processors. We are looking into improving the situation for Python 3.8.8.

Python 3.9.1 provides full support for Big Sur and Apple Silicon Macs, including building natively on Apple Silicon Macs and support for universal2 binaries.

This explains the issues. We'll keep you updated with our findings. Pyenv needs a new release for python 3.9.1 support, we'll be able to test that once we can build 3.9.1.

CleanCut commented 3 years ago

Yikes! Keep me posted!

sodul commented 3 years ago

I was able to confirm that all our python unittests are now passing with Python 3.9.1 on macOS BigSur.

While there is no new release of pyenv yet the default branch support 3.9.1. We use curl https://pyenv.run | bash which allows us to get the latest changes without having to wait for brew to be updated.

pyenv update
pyenv install 3.9.1
pyenv global 3.9.1

Then installed our dependencies from our requirements.txt then our test suites through green.

I hope this will help others.

CleanCut commented 3 years ago

Ah, so it was a pyenv-specific problem. Thanks for following up!