python / cpython

The Python programming language
https://www.python.org
Other
63.73k stars 30.53k forks source link

A bug in the multiprocessing module #82376

Open 167d1f48-81c8-4cdf-91c6-26d7097f932f opened 5 years ago

167d1f48-81c8-4cdf-91c6-26d7097f932f commented 5 years ago
BPO 38195
Nosy @pitrou, @applio, @iritkatriel
Files
  • testqueue.py: Example of a freezing program
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['extension-modules', 'type-bug', '3.9'] title = 'A bug in the multiprocessing module' updated_at = user = 'https://bugs.python.org/eshkrig' ``` bugs.python.org fields: ```python activity = actor = 'eshkrig' assignee = 'none' closed = False closed_date = None closer = None components = ['Extension Modules'] creation = creator = 'eshkrig' dependencies = [] files = ['48609'] hgrepos = [] issue_num = 38195 keywords = [] message_count = 3.0 messages = ['352613', '411500', '411645'] nosy_count = 4.0 nosy_names = ['pitrou', 'davin', 'eshkrig', 'iritkatriel'] pr_nums = [] priority = 'normal' resolution = None stage = 'resolved' status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue38195' versions = ['Python 3.9'] ```

    167d1f48-81c8-4cdf-91c6-26d7097f932f commented 5 years ago

    Hi!

    I found a bug in the multiprocessing module.

    The documentation (https://docs.python.org/3.6/library/multiprocessing.html) for the Queue.get method says: "If timeout is a positive number, it blocks at most timeout seconds ..."

    But the Queue.get method freezes if the process filling the queue terminates while reading data.

    In the attachment is an example of a program (testqueue.py) that freezes after ~ 5-30 seconds.
    After pressing <control>-<c>, the output is as follows:
    Traceback (most recent call last):
      File "./testqueue.py", line 45, in <module>
        result = ps_queues[cpu].get(timeout=2)
      File "/usr/lib64/python3.6/multiprocessing/queues.py", line 108, in get
        res = self._recv_bytes()
      File "/usr/lib64/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
        buf = self._recv_bytes(maxlength)
      File "/usr/lib64/python3.6/multiprocessing/connection.py", line 411, in _recv_bytes
        return self._recv(size)
      File "/usr/lib64/python3.6/multiprocessing/connection.py", line 379, in _recv
        chunk = read(handle, remaining)
    KeyboardInterrupt

    Please fix it.

    iritkatriel commented 2 years ago

    Python 3.6 is no longer maintained and this doesn't freeze on my Mac on 3.11.

    Please create a new issue if you are seeing the freeze on a current python version (>= 3.9) and specify which system you are running it on.

    167d1f48-81c8-4cdf-91c6-26d7097f932f commented 2 years ago
    $ cat /etc/os-release
    NAME=Gentoo
    ID=gentoo
    PRETTY_NAME="Gentoo/Linux"
    ANSI_COLOR="1;32"
    HOME_URL="https://www.gentoo.org/"
    SUPPORT_URL="https://www.gentoo.org/support/"
    BUG_REPORT_URL="https://bugs.gentoo.org/"
    
    $ uname -a
    Linux lynx 5.16.2-gentoo-1 #1 SMP PREEMPT Fri Jan 21 10:38:04 MSK 2022 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux
    
    $ python --version
    Python 3.9.9
    $ ./testqueue.py
    queue.get() from pid 11371
    queue.get() from pid 11372
    ^CTraceback (most recent call last):
      File "/home/john/main/123/./testqueue.py", line 45, in <module>
        result = ps_queues[cpu].get(timeout=2)
      File "/usr/lib/python3.9/multiprocessing/queues.py", line 117, in get
        res = self._recv_bytes()
      File "/usr/lib/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
        buf = self._recv_bytes(maxlength)
      File "/usr/lib/python3.9/multiprocessing/connection.py", line 426, in _recv_bytes
        return self._recv(size)
      File "/usr/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
        chunk = read(handle, remaining)
    KeyboardInterrupt
    eshkrig commented 1 year ago
    $ cat /etc/os-release
    NAME=Gentoo
    ID=gentoo
    PRETTY_NAME="Gentoo Linux"
    ANSI_COLOR="1;32"
    HOME_URL="https://www.gentoo.org/"
    SUPPORT_URL="https://www.gentoo.org/support/"
    BUG_REPORT_URL="https://bugs.gentoo.org/"
    VERSION_ID="2.13"
    
    $ uname -a
    Linux lynx 6.3.6-gentoo-1 #1 SMP PREEMPT_DYNAMIC Fri Jun  9 12:39:50 MSK 2023 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux
    
    $ python --version
    Python 3.11.3
    $ ./testqueue.py
    queue.get() from pid 11623
    queue.get() from pid 11624
    queue.get() from pid 11625
    ^CTraceback (most recent call last):
      File "/home/john/main/123/./testqueue.py", line 45, in <module>
        result = ps_queues[cpu].get(timeout=2)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/multiprocessing/queues.py", line 117, in get
        res = self._recv_bytes()
              ^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/multiprocessing/connection.py", line 215, in recv_bytes
        buf = self._recv_bytes(maxlength)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/multiprocessing/connection.py", line 420, in _recv_bytes
        return self._recv(size)
               ^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/multiprocessing/connection.py", line 378, in _recv
        chunk = read(handle, remaining)
                ^^^^^^^^^^^^^^^^^^^^^^^
    KeyboardInterrupt
    eshkrig commented 5 months ago
    $ cat /etc/os-release
    NAME=Gentoo
    ID=gentoo
    PRETTY_NAME="Gentoo Linux"
    ANSI_COLOR="1;32"
    HOME_URL="https://www.gentoo.org/"
    SUPPORT_URL="https://www.gentoo.org/support/"
    BUG_REPORT_URL="https://bugs.gentoo.org/"
    VERSION_ID="2.15"
    
    $ uname -a
    Linux lynx 6.6.32-gentoo-1 #1 SMP PREEMPT_DYNAMIC Fri May 31 10:05:26 MSK 2024 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux
    
    $ python --version
    Python 3.12.3
    $ ./testqueue.py
    queue.get() from pid 13813
    queue.get() from pid 13814
    queue.get() from pid 13815
    queue.get() from pid 13817
    ^CTraceback (most recent call last):
      File "/home/john/./testqueue.py", line 45, in <module>
        result = ps_queues[cpu].get(timeout=2)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.12/multiprocessing/queues.py", line 117, in get
        res = self._recv_bytes()
              ^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.12/multiprocessing/connection.py", line 216, in recv_bytes
        buf = self._recv_bytes(maxlength)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.12/multiprocessing/connection.py", line 437, in _recv_bytes
        return self._recv(size)
               ^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.12/multiprocessing/connection.py", line 395, in _recv
        chunk = read(handle, remaining)
                ^^^^^^^^^^^^^^^^^^^^^^^
    KeyboardInterrupt