Open 167d1f48-81c8-4cdf-91c6-26d7097f932f opened 5 years ago
Hi!
I found a bug in the multiprocessing module.
The documentation (https://docs.python.org/3.6/library/multiprocessing.html) for the Queue.get method says: "If timeout is a positive number, it blocks at most timeout seconds ..."
But the Queue.get method freezes if the process filling the queue terminates while reading data.
In the attachment is an example of a program (testqueue.py) that freezes after ~ 5-30 seconds.
After pressing <control>-<c>, the output is as follows:
Traceback (most recent call last):
File "./testqueue.py", line 45, in <module>
result = ps_queues[cpu].get(timeout=2)
File "/usr/lib64/python3.6/multiprocessing/queues.py", line 108, in get
res = self._recv_bytes()
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 411, in _recv_bytes
return self._recv(size)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
Please fix it.
Python 3.6 is no longer maintained and this doesn't freeze on my Mac on 3.11.
Please create a new issue if you are seeing the freeze on a current python version (>= 3.9) and specify which system you are running it on.
$ cat /etc/os-release
NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo/Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"
$ uname -a
Linux lynx 5.16.2-gentoo-1 #1 SMP PREEMPT Fri Jan 21 10:38:04 MSK 2022 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux
$ python --version
Python 3.9.9
$ ./testqueue.py
queue.get() from pid 11371
queue.get() from pid 11372
^CTraceback (most recent call last):
File "/home/john/main/123/./testqueue.py", line 45, in <module>
result = ps_queues[cpu].get(timeout=2)
File "/usr/lib/python3.9/multiprocessing/queues.py", line 117, in get
res = self._recv_bytes()
File "/usr/lib/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.9/multiprocessing/connection.py", line 426, in _recv_bytes
return self._recv(size)
File "/usr/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
$ cat /etc/os-release
NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"
VERSION_ID="2.13"
$ uname -a
Linux lynx 6.3.6-gentoo-1 #1 SMP PREEMPT_DYNAMIC Fri Jun 9 12:39:50 MSK 2023 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux
$ python --version
Python 3.11.3
$ ./testqueue.py
queue.get() from pid 11623
queue.get() from pid 11624
queue.get() from pid 11625
^CTraceback (most recent call last):
File "/home/john/main/123/./testqueue.py", line 45, in <module>
result = ps_queues[cpu].get(timeout=2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/queues.py", line 117, in get
res = self._recv_bytes()
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/connection.py", line 215, in recv_bytes
buf = self._recv_bytes(maxlength)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/connection.py", line 420, in _recv_bytes
return self._recv(size)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/connection.py", line 378, in _recv
chunk = read(handle, remaining)
^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
$ cat /etc/os-release
NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"
VERSION_ID="2.15"
$ uname -a
Linux lynx 6.6.32-gentoo-1 #1 SMP PREEMPT_DYNAMIC Fri May 31 10:05:26 MSK 2024 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux
$ python --version
Python 3.12.3
$ ./testqueue.py
queue.get() from pid 13813
queue.get() from pid 13814
queue.get() from pid 13815
queue.get() from pid 13817
^CTraceback (most recent call last):
File "/home/john/./testqueue.py", line 45, in <module>
result = ps_queues[cpu].get(timeout=2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/queues.py", line 117, in get
res = self._recv_bytes()
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/connection.py", line 437, in _recv_bytes
return self._recv(size)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/connection.py", line 395, in _recv
chunk = read(handle, remaining)
^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields: ```python assignee = None closed_at = None created_at =
labels = ['extension-modules', 'type-bug', '3.9']
title = 'A bug in the multiprocessing module'
updated_at =
user = 'https://bugs.python.org/eshkrig'
```
bugs.python.org fields:
```python
activity =
actor = 'eshkrig'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation =
creator = 'eshkrig'
dependencies = []
files = ['48609']
hgrepos = []
issue_num = 38195
keywords = []
message_count = 3.0
messages = ['352613', '411500', '411645']
nosy_count = 4.0
nosy_names = ['pitrou', 'davin', 'eshkrig', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38195'
versions = ['Python 3.9']
```