Redundant pipes

[kazu-yamamoto]

The current IO managers consume 4 file descriptors. If the timeout manager takes a role of killing IO managers, they can have only one pipe for wake up (2 file descriptors). In this case, the timeout manger also can have just one pipe for die/signal.

[AndreasVoellmy]

Let me make sure I understand the file descriptors used. For a single manager, if it is an epoll or kqueue backend, we have one file for the epoll or kqueue. Then the Control value in EventManager has two descriptors for the read and write ends of the "control" file descriptors and either one file descriptor for the controlEventFd in the case that we HAVE_EVENTFD or otherwise two file descriptors for the read and write ends of the "wake" descriptors. So, assuming epoll or kqueue and HAVE_EVENTFD we get 4 file descriptors. Is that the way you count it also?

[AndreasVoellmy]

What is your concern with this issue? Do you think 4 file descriptors per core used is too much?

Can you explain your alternative design in a bit more detail? I don't see how you can eliminate the file descriptors. You will need an eventfd for the wake up mechanism because it is fast and even with the epoll backend we need to wake it up if the manager needs to be shutdown (although that is the only use in this case, so we could reuse the control pipe for this). And you still need the control pipe, b/c the RTS uses this to send signals.

[AndreasVoellmy]

Actually, there may be a bug in the current code. I see in GHC.Event.Control that when newControl is executed it calls 'c_setIOManagerControlFd ctrl_wr' on the write end of the control pipe. This sets the control pipe fd that the RTS uses to signal the IO manager using this Control value. It looks like there is just a single global variable and since we now create one Control value per core (since one per manager and one manager per core) this variable will be overwritten and the RTS can only signal one IO manager. I'll have to look into this more to determine whether this is a problem.

[kazu-yamamoto]

In Marlow's IO manager, a pair of pipe (2 FDs) is consumed. Wakeup/Die/Signal are sent in this single channel. Bryan and Johan's IO manger, two pairs of pipe (4 FDs) are consumed. They separate a channel for Wakeup from other control messages.

Please read 6.4 of "Extending the Haskell Foreign Function Interface with Concurrency" and 8.1 of "Scalable Event Handling for GHC".

For wake-up design, we cannot reduce this channels.

But for non-wake-up design, the channel for Wakeup is not necessary for IO managers. If the timeout manager holds Haskell thread IDs of other IO managers, the timeout manager can killThread the IO managers and tear-down process can be done in their exception handler. So, only the channel for Die/Signal for the timeout manager is necessary.

I'm not sure that we can killThread the timeout manager. But if we can, no channel is necessary.

My purpose of this discussion is to remove unnecessary resources. Less resources used, the better. And code readability will be improved.

[kazu-yamamoto]

I don't count file for the epoll or kqueue. I'm just talking about two pairs of pipes. EventFD is Linux dependent. I don't know how many FDs are necessary for EventFD. But if it is 1 as you explained, three FDs are consumed on Linux.

[kazu-yamamoto]

I'm assuming non-wake-up design because I would like to switch from wake-up to non-wake-up for kqueue. FreeBSD/NetBSD/OpenBSD/Mac has kqueue. So, which platform still uses Poll backend? Windows? We should contact with Joey Adams to see if Windows provides separate methods for registration and polling.

[AndreasVoellmy]

I just pushed a patch that fixes the problem that I described above. The fix is to have the timer manager register the pipes used by its Control object with the RTS, but have the per-cap IO managers not register their control object with the RTS. Therefore all the signals sent by RTS to IO manager go to the single timer manager. If the timer manager receives a "die" message, it causes (indirectly) the other per-cap managers to shutdown. One nice thing about this design is that it avoids RTS changes.

Another nice thing is that it enables us to reduce the number of file descriptors used, as you suggested. For backends like EPOLL and your new KQueue we can eliminate the wakeup file descriptor. I have not done this yet.