reverbrain / elliptics

Distributed storage for medium and large objects with streaming support
http://reverbrain.com/elliptics
GNU Lesser General Public License v3.0
497 stars 85 forks source link

net: fix addicting net thread on sending responses to one state #712

Open shaitan opened 8 years ago

shaitan commented 8 years ago

If there are a lot of small responses which should be sent to some state with good connection, e.g. iterator generates such responses, net thread can get into cycle of sending these responses and will not send responses to other states until cycle is done. If responses is rather small, the cycle will be done only when all responses are sent.

I've added break to this cycle if a response is fully sent, so net thread will switch to another state for sending response to it.

bioothod commented 8 years ago

Should this problem be handled by having more network threads?

This patch forces elliptics to go to kernel, to reschedule process and so on - just to send a single packet. And if there are multiple packets in the queue, this will require multiple epoll schedule roundtrip.

shaitan commented 8 years ago

Should this problem be handled by having more network threads?

only if each state including accepter will have separate thread otherwise iteration spams one net thread, so it stops serving other states.

shaitan commented 8 years ago

We could add some limit on number of responses sent in a row to one state, so instead of breaking after each response it will break only if quite a lot of responses are sent to some state. But I think we should proceed from reality and measure how it affects to iteration because other commands doesn't produce such number of small responses.