Open cemysce opened 1 year ago
FYI I came across this issue because I was investigating a program I'm calling with subprocess.run
on Windows, whose stderr I'm directing to a file. (It happens to be a Python program, though I don't think that actually matters here.) The program has its own bug, whereby it prints to stderr with '\n'
line terminators, regardless of the platform on which it is running.
So I was expecting to be able to use pass the file object (which is actually a TextIOWrapper
with appropriate value for newline
) to subprocess.run
and have the line endings seamlessly translated.
Documentation
The documentation for the
subprocess
module (I'm looking at both 3.8 and 3.11) makes the following misleading or inaccurate claims (excerpts taken from 3.11, but similar statements exist in both):subprocess.run
(3.8, 3.11):io.TextIOWrapper
default. The _universalnewlines argument is equivalent to text and is provided for backwards compatibility. By default, file objects are opened in binary mode.io.TextIOWrapper
.'\n'
in the input will be converted to the default line separatoros.linesep
. For stdout and stderr, all line endings in the output will be converted to'\n'
. For more information see the documentation of theio.TextIOWrapper
class when the newline argument to its constructor isNone
.In reality, even if text mode is enabled (as required by the above statements) these translations only occur for a stream (
stdin
,stdout
, orstderr
) if that stream is specified as or implied to besubprocess.PIPE
. It's not clear to a reader that if they pass a file object (for instance aTextIOWrapper
) for a stream, the encapsulation (i.e. translation features) provided by that file object is completely circumvented becausesubprocess
actually just passes the underlying file descriptor directly to the child process.There are also issues with phrases like "file objects ... stdin, stdout and stderr ... opened in text mode" (variations of this appear a few times):
Popen.__init__
(which are actually wrapping internally-constructed pipes, not the passed-in files) that are opened in text mode. But this is an implementation detail, the documentation shouldn't get bogged down in trying to explain it. Instead it should simply state that these translation options only apply to streams which are specified assubprocess.PIPE
, and in such case (and only if in text mode) the pipes are read/written using aTextIOWrapper
with the specifiedencoding
anderrors
parameters (orTextIOWrapper
's defaults for those parameters).Just to be clear, I think the actual implementation of
subprocess
is totally reasonable. Otherwise,subprocess
would have to use a pipe under the hood. Then ifstdin
was passed a file object,subprocess
would have to read from the file object, performing appropriate translations, then write to a pipe, and pass the pipe's read fd as the child's stdin. And ifstdout
/stderr
was passed a file object,subprocess
would have to pass a pipe's write fd as the child's stdout/stderr, then read from that pipe, and perform translations when writing to the file object. All this functionality is best implemented by the application itself.My issue is just about the documentation, which implies the above functionality when in fact all translation is bypassed.