python / cpython

The Python programming language
https://www.python.org
Other
63.49k stars 30.4k forks source link

contextvars.Context.run hangs forever in ProccessPoolExecutor #83438

Open 64dd691a-4e3d-4c6f-81f2-2c6bab1e5948 opened 4 years ago

64dd691a-4e3d-4c6f-81f2-2c6bab1e5948 commented 4 years ago
BPO 39257
Nosy @1st1, @Ronserruya

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-bug', '3.8'] title = 'contextvars.Context.run hangs forever in ProccessPoolExecutor' updated_at = user = 'https://github.com/Ronserruya' ``` bugs.python.org fields: ```python activity = actor = 'yselivanov' assignee = 'none' closed = False closed_date = None closer = None components = [] creation = creator = 'ronserruya' dependencies = [] files = [] hgrepos = [] issue_num = 39257 keywords = [] message_count = 2.0 messages = ['359575', '359749'] nosy_count = 2.0 nosy_names = ['yselivanov', 'ronserruya'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue39257' versions = ['Python 3.8'] ```

64dd691a-4e3d-4c6f-81f2-2c6bab1e5948 commented 4 years ago

Sending Context.run to another process via ProccessPoolExecutor hangs forever:

from contextvars import ContextVar, copy_context
from concurrent.futures import ProcessPoolExecutor
from multiprocessing import Process

var: ContextVar[int] = ContextVar('var',default=None)

if __name__ == '__main__':

    # ***** This hangs forever *****
    with ProcessPoolExecutor(max_workers=1) as pp:
        ctx = copy_context()
        pp.submit(ctx.run, list)

    # ****** This throws 'cannot pickle Context' # ***** This hangs forever *****
    ctx = copy_context()
    p = Process(target=ctx.run, args=(list,))
    p.start()
    p.join()

python version is 3.8.0 running on Mac OSX 10.15.1

1st1 commented 4 years ago

This throws 'cannot pickle Context'

Yes, this is expected. contextvars are not compatible with multiprocessing.

** This hangs forever ***

This hanging part is weird, and most likely hints at a bug in multiprocessing.