sunmy2019
commented
1 year ago 1
In the doc it says, once EOF is received, no more data_received will be called, and connection_lost will be called on EOF. Transport itself will decide whether to close the connection or not.
These are clearly documented and widely used and implemented also in other event-loop implementations. Thus, it is impractical to change. I also intuitively against to change that.
2
In addition, I think the correct solution here is to call connect_read_pipe again with another stream reader. For example, with uvloop installed, you can write something like this. https://github.com/MagicStack/uvloop
import sys
import asyncio
import uvloop
async def read_stdin():
loop = asyncio.get_running_loop()
reader = asyncio.StreamReader()
protocol = asyncio.StreamReaderProtocol(reader)
await loop.connect_read_pipe(lambda: protocol, sys.stdin)
async for line in reader:
print("read:", line.decode())
print("EOF")
reader = asyncio.StreamReader()
protocol = asyncio.StreamReaderProtocol(reader)
await loop.connect_read_pipe(lambda: protocol, sys.stdin)
async for line in reader:
print("read:", line.decode())
uvloop.install()
asyncio.run(read_stdin())This is exactly what you have expected, since uvloop will not close the underlying file when meeting eof.
3
I think the main problem is that, _UnixReadPipeTransport holds the ownership of the pipe. As mentioned in the comment here, it seems like a design choice from 10 years ago.
I do not know why it is designed this way. (But removing all self._pipe.close() does make the above code working 🤣)
gvanrossum
micha030201
GalaxySnail
willingc
Background
An "end-of-file condition" or EOF in POSIX is what happens when a process makes a read() call on a file descriptor and gets a return value of 0. Files can sometimes continue after an end-of-file condition has been encountered.
asyncio's
connect_read_pipeon Unix can take files of three types -- FIFO (or unnamed pipe[^S_ISFIFO]), socket, and character device. EOF means different things in each of these file types, so let's go through them one by one.Socket
In docs to read():
In docs to recv():
What that means, however, varies based on the socket type. There are 4 of them: SOCK_DGRAM, SOCK_RAW, SOCK_SEQPACKET, and SOCK_STREAM[^socket_types]. Let's go through them.
SOCK_SEQPACKET and SOCK_STREAM
These are connection-mode, so when whoever we've been talking to shuts the socket down, the connection is closed and no more data will be sent.
Well, almost -- from the Linux man pages:
But provided that we don't call read() with length 0, EOF means that the connection is closed.
SOCK_DGRAM and SOCK_RAW
These, however, are connectionless-mode, so they can't be shut down from the other side, and thus we would never encounter EOF.
With datagram sockets there is a small caveat. In the Linux manual page for recv:
However, on Linux that is not a problem, because it breaks POSIX-compatibility in this instance:
So on Linux, we're still fine. On a POSIX-compatible system, we would confuse receiving a zero-length datagram with a connection being closed.
Raw socket datagrams always include the IP header, so their length can never be 0[^man_raw].
FIFO/pipe
In docs to read():
What this says is that when a reader encounters EOF on a pipe, it means that the all writers have closed their FIFO or pipe file descriptors. It will keep encountering EOF with every call to read() while there are no writers.
If it's an unnamed pipe, it means that its write end is lost forever, because it can't be referred to in any other way than its file descriptor. If it's a FIFO, however, a new process can open it for writing and our reader will stop encountering EOF and will be able to proceed as it had before.
Character device
In docs to read():
That means that Linux or BSD or whoever can figure out whatever each of them wants to do when a reader encounters EOF on a character device. I have not tested any other systems, but Linux seems to generate an EOF exactly once, and then, if the device is still operational, all subsequent reads will proceed as normal.
Application
One of the consequences of that is that programs running in a terminal can keep reading from stdin and generally doing anything they want after the user presses Ctrl+D.
Some programs take advantage of that, one example is GDB -- if you press Ctrl+D while debugging a remote inferior, it will ask for confirmation and warn you that the inferior would be detached. If you change you mind and choose no, it will continue working as it was before.
It can do that without any issue because sending an EOF on a terminal is just a way to communicate with the process reading from it, it doesn't close the file or prevent further communication.
asyncio behavior
When a
_UnixReadPipeTransportin asyncio encounters EOF, the following happens:https://github.com/python/cpython/blob/aa874326d86734606a1930e7f48e2ee204cae0b6/Lib/asyncio/unix_events.py#L527-L532
It stops reading from that file (unregisters it from selection), and calls
connection_loston the protocol.It also destroys itself and closes the file:
https://github.com/python/cpython/blob/aa874326d86734606a1930e7f48e2ee204cae0b6/Lib/asyncio/unix_events.py#L587-L594
This behavior is correct when we're dealing with stream (or technically datagram, but in that case it would just never execute) sockets or unnamed pipes, but it's incorrect if what we're reading from anything else -- a FIFO or a character device.
Consequences
It means that the kind of thing GDB does is not possible with asyncio's
connect_read_pipe. Here's an example of what doesn't work:Additionally,
stdinis now closed, so callingconnect_read_pipeagain will not work either.The only ways to work around that on asyncio would be to either monkeypatch
_UnixReadPipeTransport, or useos.dup. I suppose one could also wrap thepipefile object fed toconnect_read_pipeto return a sentinel instead of an empty bytes object on EOF, and also modify the protocol to interpret that sentinel in the correct way. None of these options are particularly pleasant.The same things works without issue with standard Python file API:
Possible solution
I don't know how Python developers feel about changing the asyncio API in ways that could subtly break programs, but I hope that the argument for correctness and the fact that some fairly important use cases are currently impossible are strong enough to warrant such a change.
Ideally,
_UnixReadPipeTransportwould be changed not to callconnection_loston EOF, instead only calling in on exception. The task of interpreting EOF then would be placed upon protocols. The list of protocols already includes specific protocols for datagrams, streams, and unnamed pipes, so it's not a huge stretch to also have specific protocols for FIFOs and character devices. Their behavior is different from all the protocols that are already there, and I would wager that reading from a terminal is a pretty common use case.Alternatively,
connect_read_pipecould return different transports for different file types, but that would require it to distinguish between unnamed pipes and FIFOs, which I'm not sure is possible.Additionally,
StreamReaderdoes not work correctly when reading from a named pipe or a character device, as it assumes that EOF is a permanent state -- which is pretty reasonable, it is a stream reader and it works for stream sockets, the issue is that a character device or pipe is not one. It could be modified to accommodate for those types of files, should the responsibility for indicating that the connection is closed when a stream socket receives an EOF be put onProtocol. That would make stream in this instance an abstraction specific to asyncio and not one that refers to a POSIX stream socket. Otherwise, other types of readers (and probably writers, I didn't even touch on those) should be created. I admit thatCharacterDeviceReaderdoes not exactly roll off the tongue.I can't really think of less drastic ways to fix this, but perhaps there are some that I'm missing. Anyway, thank you for reading this! This is the first issue I'm submitting in the Python repository and I've tried to make it as detailed as possible.
[^S_ISFIFO]: IEEE Std 1003.1-2017, Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, 2018 Edition, ISBN 978-1-5044-4542-9 -- line 13390, page 393
[^socket1]: IEEE Std 1003.1-2017, Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, 2018 Edition, ISBN 978-1-5044-4542-9 -- line 57251, page 1772
[^socket2]: IEEE Std 1003.1-2017, Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, 2018 Edition, ISBN 978-1-5044-4542-9 -- lines 58076-58078, page 1793
[^socket_types]: IEEE Std 1003.1-2017, Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, 2018 Edition, ISBN 978-1-5044-4542-9 -- section 2.10.6, lines 18376-18377, page 524
[^man_recv]: recv(2), Linux man-pages 6.04
[^man_raw]: raw(7), Linux man-pages 6.04
[^fifo]: IEEE Std 1003.1-2017, Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, 2018 Edition, ISBN 978-1-5044-4542-9 -- lines 57221-57227, page 1771
[^char_device]: IEEE Std 1003.1-2017, Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, 2018 Edition, ISBN 978-1-5044-4542-9 -- lines 57217, page 1771