proposal: os: API to detect that the read end of a pipe was closed

[ibukanov]

Currently there is no API in Go to detect that the read end of a pipe was closed without writing to the pipe a non-empty slice. This prevents, for example, to write in a safe Go a version of GNU tail utility that exits immediately when it detects that the read end of its stdout was closed even when it waits for more input like when it is called via tail -f. For example, given the following case:

tail -f file-to-follow | grep -q foo

GNU tail exits immediately after the grep finds the word foo and terminates without waiting that the file-to-follow will be extended and trying to write those extra bytes to stdout.

Yet to write such functionality in Go one needs to use unsafe code to call sys.Select or similar and wait for an syscall.EPIPE from the stdout descriptor.

What version of Go are you using (go version)?

go1.10.3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

amd64 linux fedora-28

[ianlancetaylor]

Do you have a proposal for how to address this?

[ibukanov]

An API that is sufficient to implement that tail-like functionality is to add to os.signal something like

func ClosedPipe(pipe *os.File) (<-chan struct{}, error)

that returns a channel that becomes ready when the read end of the pipe closes. It is very OK if this only works with pipes where SetWriteDeadline works, i.e. the pipes that the poller supports.

Alternatively the API can take a callback that is called when the pipe is closed, i.e. like

func ClosedPipe(pipe *os.File, callback func()) error

The nice thing of this form is that one can pass there CancelFunc from context.WithCancel() as a typical reaction to the closed read end of the pipe is to stop current activities that in future result in writes to the pipe.

[robpike]

How do I detect that the read end has been closed?

[ibukanov]

@robpike On Linux when the read end is closed poll/epoll syscalls for the write end of the pipe return POLLERR/EPOLLERR.

[cespare]

@robpike was this close intentional?

[ibukanov]

On systems with kqueue the interface also provides notifications about the closed read end of the pipe. But on Windows it seems with anonymous pipes the only way to support this is via periodic calls to the write function with zero-length buffer. At least the question on StackOverflow has not got a better answer, https://stackoverflow.com/questions/11564166/detecting-if-anonymous-pipe-is-writable-to-detect-when-to-end-process

[robpike]

Almost 50 years on and we still don't understand how to handle a closed pipe. It's remarkable how much energy has been consumed on this topic, how many crashes caused, how many solutions proposed.

I would be more comfortable about this if there was a non-epoll, non-signal-catching way to learn about a closed pipe. Is there?

[ibukanov]

@robpike there is no such API besides listening for epoll/poll/kqueue events.

[robpike]

@ibukanov I am not surprised. As I said, 50 years on....

[ianlancetaylor]

Can some API along these lines work for streaming network connections as well?

I don't see why this should be a function, it seems like it ought to be a method on the *os.File or net.TCPConn. I don't know if sending a value on a channel is right, but I don't have any better suggestion. I don't think a callback is a good idea.

If this really can't be implemented on Windows, that sounds like a real problem for the whole idea.

[ibukanov]

In principle this can be implemented on Windows via periodic zero-length writes to the pipe. This is wasteful, but at least it is doable if one needs it.

[ianlancetaylor]

Zero-length writes to the pipe should also work on Unix systems.

[ianlancetaylor]

That is, if the application wants to find this out with some work, it can do zero-length writes. I don't think the Go runtime should be doing zero-length writes.

[powerman]

Actually writing empty slice does not work - it returns no error even after pipe reader exited.