replace zloop reactor

[garlick]

There are a few problems with the zloop reactor that underpins the flux reactor:

• A timer handler cancelling a poller won't prevent the poller handler from being run after the timer handler returns, provided the poller was ready in the same trip through the zloop. This is just a zloop bug that is easy to fix (see issue #135)
• A handler (timer or poller) that consumes a message from a socket belonging to another poller that was ready in the same trip through the zloop will not prevent the second socket's poller handler from being called, ready or not. This is arguably incorrect handler behavior, but behavior we need due to the next item (see issue #135).
• There is no way to prioritize one handler above another. We use an inproc socket for the putmsg queue in api and module handle implementations, and call the heavyweight zloop_poller() and zloop_poller_end() functions to force putmsg to be emptied before reading from the main sockets again.
• No handles for pollers makes canceling them inefficient and awkward.
• Odd zloop_set_tolerant() error management strategy, applies to file descriptor pollers only.

[garlick]

One option is to replace zloop with libev.

The libev priority model does not fit the putmsg use case, however libev allows watchers to be temporarily started and stopped in a lightweight manner so the current handle switching logic could work more efficiently. In fact libev has idle watchers that could allow the putmsg queue to be implemented with a simple zlist_t instead of an inproc socket.

libev can be embedded in src/common/libev if we prefer not to create a dependency on the libev package.

Should we pursue coroutines as discussed in issue #126, probably either zloop or libev will not get in the way since (using parlance of @trws's qthreads based coroutine shim) the RPC code would call coroutine_yield() and the flux reactor code wrapping every handler would call call_couroutine() to start/resume a handler.

[trws]

Just to put it out there, libuv is effectively a descendant of libev in a number of respects, and offers portability and a unified concept of threads in case we ever need to run on a non-linux platform. It offers the same idle watchers, and also has a work-queue available to offload potentially blocking operations out of the event loop thread into a worker thread. All that said, it is not as simple to embed as libev, and as far as I can tell it does not have a priority mechanism, I also don't know if the libuv concept of a poller would work for what we need.

Any of these would be agnostic to the use, or disuse, of stackful coroutines like in #126. That said, picking one that would allow us to not use the coroutines, or at least to fall back to stackless coroutines, would be more efficient.

[garlick]

As a data point, both libev and libuv have packages built for RHEL6, which makes our lives easier here and at many labs.

Name        : libuv                        Relocations: (not relocatable)
Version     : 0.10.29                           Vendor: Fedora Project
Release     : 1.el6                         Build Date: Tue 18 Nov 2014 05:52:45 PM PST
Install Date: Tue 10 Feb 2015 04:09:58 PM PST      Build Host: buildvm-03.phx2.fedoraproject.org
Group       : Development/Tools             Source RPM: libuv-0.10.29-1.el6.src.rpm
Size        : 120616                           License: MIT and BSD and ISC
Signature   : RSA/8, Tue 18 Nov 2014 09:08:51 PM PST, Key ID 3b49df2a0608b895
Packager    : Fedora Project
URL         : http://nodejs.org/
Summary     : Platform layer for node.js
Description :
libuv is a new platform layer for Node. Its purpose is to abstract IOCP on
Windows and libev on Unix systems. We intend to eventually contain all platform
differences in this library.

Name        : libev                        Relocations: (not relocatable)
Version     : 4.03                              Vendor: Fedora Project
Release     : 3.el6                         Build Date: Sat 16 Jul 2011 03:24:36 AM PDT
Install Date: Mon 05 Jan 2015 02:20:45 PM PST      Build Host: x86-02.phx2.fedoraproject.org
Group       : System Environment/Libraries   Source RPM: libev-4.03-3.el6.src.rpm
Size        : 154198                           License: BSD or GPLv2+
Signature   : RSA/8, Fri 22 Jul 2011 04:28:28 AM PDT, Key ID 3b49df2a0608b895
Packager    : Fedora Project
URL         : http://software.schmorp.de/pkg/libev.html
Summary     : High-performance event loop/event model with lots of features
Description :
Libev is modeled (very loosely) after libevent and the Event Perl
module, but is faster, scales better and is more correct, and also more
featureful. And also smaller.

[garlick]

We switched over to libev, closing.