Consider use of multiple threads

[markpapadakis]

In practice, this will only benefit latency if we are going to be dealing with a lot of compressed batch message sets, which could keep a thread busy for a long time, blocking it from processing incoming input and scheduling outgoing requests.

It should be fairly trivial to implement this, though we need to be careful about retain/release semantics, and about costs of shared access. We should probably delay this until we absolute need it, so that we 'll be better informed, and have a more evolved codebase.

[markpapadakis]

Actually, that's silly. Tank (service) doesn't compress or decompress anything whatsoever (the client does) so the above remarks are mostly irrelevant, and the case for multiple threads is very weak.

[markpapadakis]

• [x] We should actually remove all locks and atomic ops from service.

[markpapadakis]

We may actually have to use multiple threads anyway, because both sendfile() and writev() may block. See #14 If we can implement an asynchronous I/O scheme where we won't both for either calls, we can and should stick to the single thread design for simplicity.

[markpapadakis]

See https://github.com/phaistos-networks/TANK/issues/14#issuecomment-232483098 for writev() timings. Considering that, and the cost for blocking sendfile() calls, it would make a lot of sense to use threads, eventually.

[markpapadakis]

We are talking microseconds here, and I am on the fence about multiple threads. It should be trivial to do this anyway, just think we 'll get some performance drop due to cache coherency semantics and kernel-levels serialisation to FDS and other resources.

[markpapadakis]

A forthcoming update (TANK2) will rely on c++20 coroutines and io_uring for disk I/O and other blocking operations.

[markpapadakis]

We will use coroutines for each consume request, so that we suspend the coroutine if the await_ready() of the awaitable returned by Service::open_partition_log() returns false.

open_partition_log() is the only function that may block for a "a long time", depending on the structure of the partition, so ti should be very beneficial.