Consider ways to implement connection priority

[gavin-norman-sociomantic]

In a live system, some applications which access a swarm cluster are super important and need a response as soon as possible, while others are of low importance and do not require an immediate response. Currently, the nodes handle all request equally, which can lead to bursts of requests from low-importance apps overwhelming the normal requests from high-importance apps.

It should be possible to implement a priority system, whereby requests from certain clients are handled with greater priority than requests from others.

[gavin-norman-sociomantic]

@david-eckardt-sociomantic , @nemanja-boric-sociomantic , and I discussed this briefly. The most obvious way to implement this would be to handle firing socket read events in epoll more or less frequently, based on the priority of a connection.

[gavin-norman-sociomantic]

I wonder if that kind of approach would really solve the problem we have, though. The specific example we have is:

1. A real-time application performing a continuous stream of requests, which needs a guaranteed fast response time (or will time out the request on the client side).
2. A batch application which runs once a day and performs a very large bulk of requests which have no real time requirements.

A system based on epoll handling, as described above, would probably only mitigate the problem (by making the real-time app's requests handled twice as often as those of the batch app), not fix it at its root.

[nemanja-boric-sociomantic]

Having a send/recv loop in a separate thread would make sure that everything is sent/received as soon as possible, but there's a problem how to resume tasks from a different thread (otherwise we're still stuck at the epoll cycle for the synchronization), + it would be very nontrivial to get that done at this stage (where the swarm core is more or less stable).