Idle: use pipes instead of sockets to talk with user subprocess

[terryjreedy]

BPO	18823
Nosy	@terryjreedy, @ezio-melotti, @asvetlov, @serhiy-storchaka
Files	Carlson.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['expert-IDLE', 'type-bug', '3.10'] title = 'Idle: use pipes instead of sockets to talk with user subprocess' updated_at = user = 'https://github.com/terryjreedy' ``` bugs.python.org fields: ```python activity = actor = 'terry.reedy' assignee = 'none' closed = False closed_date = None closer = None components = ['IDLE'] creation = creator = 'terry.reedy' dependencies = [] files = ['34654'] hgrepos = [] issue_num = 18823 keywords = [] message_count = 6.0 messages = ['196055', '215062', '215069', '215123', '336154', '370832'] nosy_count = 5.0 nosy_names = ['terry.reedy', 'ezio.melotti', 'asvetlov', 'tshepang', 'serhiy.storchaka'] pr_nums = [] priority = 'normal' resolution = None stage = 'needs patch' status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue18823' versions = ['Python 3.10'] ```

[terryjreedy]

Idle once used os.spawnv to open the user subprocess and 2.7 still does. Sockets were then used to interact with the subprocess. Since rev71746, 2011-8-3, 3.2+ use subprocess.Popen but still use sockets. A rare but continual problem is failure to make the socket connection.

In bpo-16123, Amaury Forgeot d'Arc suggested "pipes?" The idea resurfaced again in current Idle-sig thread "Idle does not open on mac" (because socket connection timed out). Guido, who designed the current 2.x system, chimed in with "I would recommend trying to use the subprocess module."

A current python-list thread indicates that there can be problems with "Running a command line program and reading the result as it runs" http://mail.python.org/pipermail/python-list/2013-August/654265.html

In particular, "for line in p:" gets lines delayed (buffered) while Peter Otten discovered that "for line in iter(p.stdout.readline, ''):" gets them as they are produced. http://mail.python.org/pipermail/python-list/2013-August/654330.html

We first need to experiment running a simple echo server with python(w).exe. I have not succeeded yet on Windows.

[terryjreedy]

In the thread that started here https://mail.python.org/pipermail/python-dev/2014-March/133492.html in this message. https://mail.python.org/pipermail/python-dev/2014-March/133671.html Josiah Carlson posted a pastebin link about how to actually use the pipes. Guido recommended that the he post a revised version to a new issue, as a patch to subprocess, but I am uploading it here to make sure it does not get lost.

[terryjreedy]

https://mail.python.org/pipermail/python-dev/2014-March/133641.html from Victor Stinner gives example code using asyncio

[terryjreedy]

Pydev, Re: Status of PEP-3145 - Asynchronous I/O for subprocess.popen; Antoine Pitrou:

Why don't you use multiprocessing or concurrent.futures? They have everything you need for continuous conversation between processes.

[terryjreedy]

bpo-34313 reports problems with multiprocessing when using at least some versions of tkinter on some versions of macOS.

[terryjreedy]

Problems with multiprocessing and tkinter on Mac might have been due to using the default start method, which used to be fork and is now spawn. This should be retested sometime.

[arhadthedev]

Problems with multiprocessing and tkinter on Mac might have been due to using the default start method, which used to be fork and is now spawn.

• gh-84559.