search

ionelmc / python-tblib

Serialization library for Exceptions and Tracebacks.
BSD 2-Clause "Simplified" License
164 stars 33 forks source link

Possible deadlock with Python 3.8.0a3 #40

Closed hroncok closed 5 years ago

hroncok commented 5 years ago

I add this:

diff --git a/tox.ini b/tox.ini
index fba49d1..0125838 100644
--- a/tox.ini
+++ b/tox.ini
@@ -17,6 +17,7 @@ basepython =
     py35: {env:TOXPYTHON:python3.5}
     {py36,docs,spell}: {env:TOXPYTHON:python3.6}
     py37: {env:TOXPYTHON:python3.7}
+    py38: {env:TOXPYTHON:python3.8}
     {bootstrap,clean,check,report,codecov,coveralls}: {env:TOXPYTHON:python3}
 setenv =
     PYTHONPATH={toxinidir}/tests

And run:

$ tox -e py38
GLOB sdist-make: .../python-tblib/setup.py
py38 inst-nodeps: .../python-tblib/.tox/.tmp/package/1/tblib-1.4.0.zip
py38 installed: atomicwrites==1.3.0,attrs==19.1.0,Automat==0.7.0,constantly==15.1.0,coverage==4.5.3,hyperlink==19.0.0,idna==2.8,incremental==17.5.0,more-itertools==7.0.0,pluggy==0.9.0,py==1.8.0,PyHamcrest==1.9.0,pytest==4.4.1,pytest-cov==2.7.1,pytest-travis-fold==1.3.0,six==1.12.0,tblib==1.4.0,Twisted==19.2.0,zope.interface==4.6.0
py38 run-test-pre: PYTHONHASHSEED='3425619301'
py38 runtests: commands[0] | py.test --cov=tblib --cov-report=term-missing -vv tests README.rst
============================= test session starts ==============================
platform linux -- Python 3.8.0a3, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- .../python-tblib/.tox/py38/bin/python
cachedir: .tox/py38/.pytest_cache
rootdir: .../python-tblib, inifile: setup.cfg
plugins: travis-fold-1.3.0, cov-2.7.1
collected 2 items                                                              

tests/test_issue30.py::test_30 PASSED                                    [ 50%]
README.rst::README.rst PASSED                                            [100%]

(hangs here) ^C

Exception ignored in: <Finalize object, dead>
Traceback (most recent call last):
  File "/usr/lib64/python3.8/multiprocessing/util.py", line 189, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 689, in _terminate_pool
    cls._help_stuff_finish(inqueue, task_handler, len(pool))
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 674, in _help_stuff_finish
    inqueue._rlock.acquire()
KeyboardInterrupt

----------- coverage: platform linux, python 3.8.0-alpha-3 -----------
Name                            Stmts   Miss Branch BrPart     Cover   Missing
------------------------------------------------------------------------------
src/tblib/__init__.py             125     12     44      6    86.98%   12-13, 16, 88, 90, 107, 136-142, 223, 15->16, 87->88, 89->90, 98->107, 195->197, 206->223
src/tblib/cpython.py               38     10     12      6    68.00%   21, 36-39, 59, 62-63, 65, 78-79, 20->21, 35->36, 58->59, 61->62, 64->65, 76->80
src/tblib/decorators.py            26      0      0      0   100.00%
src/tblib/pickling_support.py      17      0      0      0   100.00%
------------------------------------------------------------------------------
TOTAL                             206     22     56     12    85.50%

=========================== 2 passed in 3.66 seconds ===========================
Process ForkPoolWorker-13:
Process ForkPoolWorker-16:
Process ForkPoolWorker-14:
Process ForkPoolWorker-12:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.8/multiprocessing/queues.py", line 355, in get
    with self._rlock:
  File "/usr/lib64/python3.8/multiprocessing/queues.py", line 355, in get
    with self._rlock:
Process ForkPoolWorker-9:
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib64/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib64/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib64/python3.8/multiprocessing/queues.py", line 355, in get
    with self._rlock:
  File "/usr/lib64/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib64/python3.8/multiprocessing/queues.py", line 355, in get
    with self._rlock:
  File "/usr/lib64/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Process ForkPoolWorker-11:
Traceback (most recent call last):
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib64/python3.8/multiprocessing/queues.py", line 355, in get
    with self._rlock:
  File "/usr/lib64/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Process ForkPoolWorker-10:
Traceback (most recent call last):
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib64/python3.8/multiprocessing/queues.py", line 356, in get
    res = self._reader.recv_bytes()
  File "/usr/lib64/python3.8/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib64/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib64/python3.8/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib64/python3.8/multiprocessing/queues.py", line 355, in get
    with self._rlock:
  File "/usr/lib64/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Process ForkPoolWorker-15:
Traceback (most recent call last):
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/lib64/python3.8/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib64/python3.8/multiprocessing/queues.py", line 355, in get
    with self._rlock:
  File "/usr/lib64/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
ERROR: KEYBOARDINTERRUPT
ERROR: keyboardinterrupt
ionelmc commented 5 years ago

Oooof ... it's those damn doctests using multiprocessing. I'll have to move them to a real test file.

ionelmc commented 5 years ago

It would look like mp is lil bit broken in 3.8 - atexit handler gets stuck on unclosed pool.