Open Zhou-Ao opened 3 years ago
(I just edited your comment to get syntax highlighting, didn't change content at all)
Thanks for opening the issue! It's very nice to have the sample program, too.
I'm not sure exactly what's going on, but I think it's some unfortunate interaction between pub/sub semantics and the GIL. With no sleep, the sending thread never gives up the GIL, and that means the receiving thread never has a chance to wake up and receive it. The receiving socket will then have a delayed message by the time the sending thread yields the GIL, after dropping a bunch of messages. On my computer, without a sleep, about 93% of the messages are dropped.
Whenever there is no sleep at all, the latency on my computer is even worse than on yours: a mean of 0.5 seconds, and max of 1.17 seconds.
I am testing Nanomsg NNG PUB/SUB socket performance with pynng .
In the test program below, SUB socket is used as a server to receive messages and PUB socket is used as a client to receive messages. The content of a message is the simply time when it is sent by the PUB client. The SUB server measures the latency by comparing between the time when the message is received time and the time (which is the message sent time) inside the message.
For 1,000 messages sent over ~1 second, I find two issues from the performance test: 1) The latency is very high between PUB and SUB sockets in couple of milliseconds. However, under such low load, I would expect the latency to be in scale of microseconds.
2) The latency performance improves if I make the PUB client sleep in a very short period of time. I understand that sleeping would reduce the load. But I would expect it makes not much difference given the low load. I am surprised that the latency can shoot up to 100ms if there is no sleep.
I am not sure if this issue relates to python GIL. Will my main PUB client thread compete against NNG internal I/O thread for GIL without a sleep?
Any ideas will be welcome and appreciated.
The testing programs are as below:
And the performance results on my PC are below: