subprocess line-buffering only works in universal newlines mode

[pitrou]

BPO	21471
Nosy	@gpshead, @pitrou, @benjaminp, @4kir4, @hynek

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-bug', 'library'] title = 'subprocess line-buffering only works in universal newlines mode' updated_at = user = 'https://github.com/pitrou' ``` bugs.python.org fields: ```python activity = actor = 'akira' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = creator = 'pitrou' dependencies = [] files = [] hgrepos = [] issue_num = 21471 keywords = [] message_count = 4.0 messages = ['218257', '218259', '218260', '218263'] nosy_count = 7.0 nosy_names = ['gregory.p.smith', 'pitrou', 'benjamin.peterson', 'stutzbach', 'Arfrever', 'akira', 'hynek'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue21471' versions = ['Python 3.5'] ```

[pitrou]

The docs for subprocess.Popen seem to imply that line-buffering is always available. However, bufsize=1 is a special value only when open the pipes in text mode, i.e. when "universal newlines" are enabled.

In the short term, we should probably fix the subprocess docs. In the long term, perhaps we can add a line buffering capability to BufferedWriter?

[pitrou]

Ha, it seems actually worse than that, since no buffering argument is ever passed to the TextIOWrapper constructor. "bufsize=1" will simply get ignored, and line buffering doesn't work at all.

Example under 2.7:

$ python -c 'import subprocess; p = subprocess.Popen(["/bin/cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=1, universal_newlines=True); p.stdin.write("foo\n"); print(p.stdout.readline()); p.stdin.close()'
foo

Under 3.4, the same line hangs in the p.stdout.readline() call.

[pitrou]

Ok, the "not working at all" part is issue bpo-21332. Let's make this issue specific to binary mode, again.

[7fe5d93b-2a2c-46a0-b5cd-5602c591856a]

Until the current patch for issue bpo-21332 is committed; bufsize=1 is equivalent to bufsize=-1 in both binary and text mode.

You are correct the patch in bpo-21332 fixes only the text mode (universal_newlines=True) -- it also updates the documentation to mention that bufsize=1 works only in text mode.