Closed GermanCoding closed 1 year ago
The current implementation of flume has receivers make claims on the queue, so starvation/unfairness should not occur. Threads will be woken to receive items in the approximate order that they called .recv()
.
Sounds good, that's exactly what I'm looking for! Thanks
So, I have the following use case:
I have multiple async workers (futures) that all want to process data (send over a network connection in this case).
I want to use flume to let them all receive the same set of messages, where flume picks a receiver which then performs the sending over the network.
What I'm wondering is, how fair is the load distributed across multiple receivers? In my case, when the receiver has processed data, the task isn't actually completed yet: Network data is still in-flight and needs to be processed by the other side. Yet the receiver is already capable of taking in new messages and hence will call recv() again. In this case, I want the other receivers to do work next, before the previous receiver gets new work. This would ensure equal load on all network connections.
Therefore, my question is: Does flume distribute fairly? If a receiver gets a message and then calls recv() again, can it get new work before another receiver, which has been blocked on recv() the entire time, got its share of data?