python / cpython

The Python programming language
https://www.python.org
Other
62.31k stars 29.93k forks source link

spawn asyncio subprocesses in a thread pool #81444

Open asvetlov opened 5 years ago

asvetlov commented 5 years ago
BPO 37263
Nosy @asvetlov, @1st1

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['expert-asyncio', 'performance'] title = 'spawn asyncio subprocesses in a thread pool' updated_at = user = 'https://github.com/asvetlov' ``` bugs.python.org fields: ```python activity = actor = 'iritkatriel' assignee = 'none' closed = False closed_date = None closer = None components = ['asyncio'] creation = creator = 'asvetlov' dependencies = [] files = [] hgrepos = [] issue_num = 37263 keywords = [] message_count = 1.0 messages = ['345472'] nosy_count = 2.0 nosy_names = ['asvetlov', 'yselivanov'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'performance' url = 'https://bugs.python.org/issue37263' versions = [] ```

asvetlov commented 5 years ago

Subprocess starting can be a long blocking operation, see https://github.com/python-trio/trio/issues/1109 for discussion

kumaraditya303 commented 1 year ago

From the Trio issue, this seems mostly a Windows issue where process creation is indeed very slow. So maybe this should be only done for Windows.

gvanrossum commented 1 year ago

I'm just realizing that when we use the ThreadedChildWatcher, there's a thread created for the process anyway, so maybe we could move the subprocess operation into that thread... (Then again on Windows we don't use child watchers.)

willingc commented 2 months ago

@kumaraditya303 Flagging this as an FYI as you look at the work for https://github.com/python/cpython/issues/120804. After that work is completed, we can revisit.