This is one of the remaining performance bottlenecks. Instead of busy-polling message consumers and producers in MessageDispatcher, we should do it in the NIO loop (aka DataReceivingLoop) based on the readiness flags of selected keys. I.e.:
If the key is ready for reading, then read all data and feed received messages to consumers for the corresponding peer immediately. (Currently, we read data only until the bip buffer in InboundMessageProcessor can't be wrapped around due to pending blocks; messages are stored in a queue until the MessageDispatcher consumes them on the next loop iteration, possibly after several milliseconds of sleep; read handles for pending blocks are registered in the BufferedDataRegistry and, again, consumed by the MessageDispatcher asynchronously.)
If the key is ready for writing, then poll all producers for the corresponding peer and write messages into the socket buffer. (Currently, this is done in the same fashion, but from the MessageDispatcher's thread, which leads to high lock contention on the peer's SocketChannelImpl with NIO thread, which tries to read from the same channel at the same time.)
Expected benefits:
no delay between receiving message and processing it
no need to keep the inbound network buffer full due to not yet consumed data
no lock contention on the socket channel's read and write ops
This is one of the remaining performance bottlenecks. Instead of busy-polling message consumers and producers in MessageDispatcher, we should do it in the NIO loop (aka DataReceivingLoop) based on the readiness flags of selected keys. I.e.:
If the key is ready for reading, then read all data and feed received messages to consumers for the corresponding peer immediately. (Currently, we read data only until the bip buffer in InboundMessageProcessor can't be wrapped around due to pending blocks; messages are stored in a queue until the MessageDispatcher consumes them on the next loop iteration, possibly after several milliseconds of sleep; read handles for pending blocks are registered in the BufferedDataRegistry and, again, consumed by the MessageDispatcher asynchronously.)
If the key is ready for writing, then poll all producers for the corresponding peer and write messages into the socket buffer. (Currently, this is done in the same fashion, but from the MessageDispatcher's thread, which leads to high lock contention on the peer's SocketChannelImpl with NIO thread, which tries to read from the same channel at the same time.)
Expected benefits: