solaris event ports are broken with FIFOs

[jeffpc]

It turns out that Solaris event ports (used by illumos-based distros as well as Solaris) are slightly broken for FIFOs.

Specifically, it is possible for port_getn to return an event on a FIFO fd (registered via PORT_SOURCE_FD and POLLIN) even though there is nothing available for reading. This is an illumos bug - not a powerdns bug.

Here is a minimal test case: port-test.txt.

If you run it with stdin being a file or a tty, port_getn blocks until bytes are available for reading:

$ ./a.out
port = 3, errno = 0
ret = 0, errno = 0
ret = -1, errno = 48
ucred = 803bc88
<waiting>

If, however, stdin is a pipe, port_getn returns immediately because the fd is marked as readable by the getpeerucred call.

$ cat | ./a.out
port = 3, errno = 0
ret = 0, errno = 0
ret = -1, errno = 48
ucred = 803e258
ret = 0, errno = 48
ev.portev_source = 4
ev.portev_events = 1
ev.portev_object = 0
ev.portev_user = 0

In the case of the recursor, a spurious event causes the thread to attempt to read the pipe, which causes it to block. A spurious event can be generated using the pfiles(1) utility. (e.g., pfiles 123 if 123 is the pdns_recursor pid)

Possible workarounds for this issue

1. Don't use event ports: Unfortunately, event ports are the preferred method of event monitoring. (devpoll or select are both available)
2. Test whether port_getn returns spurious events, and if so fail PortsFDMultiplexer initialization. @Habbie indicated that this will cause the daemon to fall back to a different multiplexer.
3. Convert FIFO fds to use non-blocking I/O. If the FIFO fds were non-blocking then the spurious wakeup would cause a no-op read(2) and an immediate return to port_getn to get more events.

Other systems that may be affected: OpenSolaris, Solaris 10, Solaris 11

Finally, powerdns should inspect the returned event for sanity. There are four members in the port_event_t struct:

         int       portev_events;   /* detected events           */
         ushort_t  portev_source;   /* event source              */
         uintptr_t portev_object;   /* specific to event source  */
         void      *portev_user;    /* user defined cookie       */

But powerdns only looks at one: portev_object. The portev_events should contain POLLIN and/or POLLOUT and portev_source should be PORT_SOURCE_FD.

Gory details

On illumos, FIFOs are implemented using the fifofs module which presents a filesystem that manages all FIFOs on the system. Each FIFO can be in one of two modes: fastpath or streams. In the fastpath mode, a lot of the heavy-weight STREAMS code is bypassed, however not all operations are supported. In the STREAMS mode, the full functionality is available. FIFOs are created in the fastpath mode, and if they attempt to use "advanced" functionality they are converted to streams mode. Once in a streams mode, there is no way to go back to fastpath mode.

The getpeerucred call (which is invoked by pfiles) ends up issuing an ioctl on the FIFO backed by the fifofs module. Since the _I_GETPEERCRED command is not handled by fastpath mode, the FIFO is converted to a streams mode FIFO. During this conversion, all poll and event ports waiters are awoken to cause them to re-register with the newly converted FIFO. The kernel stack:

              genunix`port_send_event+0x131
              genunix`pollwakeup+0x86
              genunix`strpollwakeup+0x20
              fifofs`fifo_fastturnoff+0xae
              fifofs`fifo_fastoff+0x92
              fifofs`fifo_fastioctl+0x19d
              fifofs`fifo_ioctl+0x3d
              genunix`fop_ioctl+0x55
              genunix`getpeerucred+0xff
              genunix`ucredsys+0x52
              genunix`ucredsys32+0x1b
              unix`sys_syscall32+0xff

pfiles(1) works by injecting syscalls into the target process. One of the syscalls that it injects is the call to getpeerucred. This is why pfiles(1) cat trigger this issue.

[cmouse]

:+1: for issue =)

[ghost]

:clap:

[Habbie]

port-test brokenness confirmed on:

SunOS unstable10x 5.10 Generic_147441-19 i86pc i386 i86pc
SunOS unstable10s 5.10 Generic_150400-17 sun4v sparc SUNW,SPARC-Enterprise-T5220
SunOS unstable11x 5.11 11.2 i86pc i386 i86pc
SunOS unstable11s 5.11 11.2 sun4u sparc SUNW,SPARC-Enterprise

[jeffpc]

For the record, here is the illumos bug about this: https://www.illumos.org/issues/6474

[binarycrusader]

The bug is already resolved in the next release of Oracle Solaris and is tracked under bug 17943298.