By default Go sets O_CLOEXEC on file descriptors it creates. This ensures that they aren't accidentally inherited over fork+exec.
When binding a port on the host we first try net.Listen. On modern macOS this works for
unprivileged ports (> 1024)
privileged ports on INADDR_ANY (all IPs)
Unfortunately it doesn't work for privileged ports on specific IPs, such as 127.0.0.1:80. For these cases we contact a helper process which runs as root, which binds the port and then uses sendmsg to transmit it to us. Unfortunately when we recvmsg a file descriptor, the resulting descriptor does not have O_CLOEXEC set, so it is captured by future subprocesses. Even if the parent process calls close(), the port is still in use by the subprocesses and can't be re-bound. This caused the failure seen in https://github.com/docker/for-mac/issues/5649
The solution is to ensure all file descriptors have O_CLOEXEC set. On Linux there's a sendmsg flag to set the flag atomically on received fds, but this is missing on Darwin. The best we can do is to call fcntl to set the flag ourselves after receiving the fd. Unfortunately this means there's a small race window where a fork would inherit the fd, but this shouldn't happen very often in practice.
Go will call fcntl for us if we "adopt" the file descriptor using os.NewFile and then use net.FileListener and net.FilePacketConn. This allows us to remove a lot of complicated wrapping code and allows us to remove a workaround for Close hanging when blocked in Accept, caused by the state of the file descriptor not being correct. This required some minor changes to types (net.Listener rather than *net.TCPListener) but it's worth it.
It's possible to work around this problem in other ways in the client code, for example:
always speculatively closing file descriptors after fork but before exec. This can be expensive with high ulimit values, if there is no API to enumerate open fds. I'm not sure it can be done easily in Go where the exec.Command API is high-level.
ban running long-lived subprocesses in your program. Restructure the code so that the long-lived subprocesses are managed by a separate process which doesn't bind ports.
By default Go sets
O_CLOEXECon file descriptors it creates. This ensures that they aren't accidentally inherited over fork+exec.When binding a port on the host we first try
net.Listen. On modern macOS this works forINADDR_ANY(all IPs)Unfortunately it doesn't work for privileged ports on specific IPs, such as
127.0.0.1:80. For these cases we contact a helper process which runs as root, which binds the port and then usessendmsgto transmit it to us. Unfortunately when werecvmsga file descriptor, the resulting descriptor does not haveO_CLOEXECset, so it is captured by future subprocesses. Even if the parent process callsclose(), the port is still in use by the subprocesses and can't be re-bound. This caused the failure seen in https://github.com/docker/for-mac/issues/5649The solution is to ensure all file descriptors have
O_CLOEXECset. On Linux there's asendmsgflag to set the flag atomically on received fds, but this is missing on Darwin. The best we can do is to callfcntlto set the flag ourselves after receiving the fd. Unfortunately this means there's a small race window where a fork would inherit the fd, but this shouldn't happen very often in practice.Go will call
fcntlfor us if we "adopt" the file descriptor usingos.NewFileand then usenet.FileListenerandnet.FilePacketConn. This allows us to remove a lot of complicated wrapping code and allows us to remove a workaround forClosehanging when blocked inAccept, caused by the state of the file descriptor not being correct. This required some minor changes to types (net.Listenerrather than*net.TCPListener) but it's worth it.It's possible to work around this problem in other ways in the client code, for example:
forkbut beforeexec. This can be expensive with highulimitvalues, if there is no API to enumerate open fds. I'm not sure it can be done easily in Go where theexec.CommandAPI is high-level.Signed-off-by: David Scott dave@recoil.org