set TCP_NODELAY

[mdavidsaver]

Turns out, this flag can actually reduce latancy in some situations where Nagle interacts with delayed ack. While I (and others) think it may be preferable to disable delayed ack., there is no portable way to do so. So instead, do as CA and pvAccessCPP do and set TCP_NODELAY.

https://en.wikipedia.org/wiki/Nagle%27s_algorithm https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment

[mdavidsaver]

I admit that was a bit skeptical as to whether TCP_NODELAY could have much effect on CA or PVA traffic with current IP stacks. However, one use case has been identified which does trigger extra latency. Executing PUT to a single PV in a tight loop. Presumably this would also effect GET and RPC operations.

Note, times shown in the following wireshark output are relative to the previous displayed frame.

Filtering first on pva shows an extra ~40ms between the client sending DESTROY_REQUEST of a previous request, and sending the creation request for the next. On reflection, this is a fine example of the send -> send -> recv sequence which is known to trigger a conflict between nagle and delayed ack.

No. Time    Source  Destination Protocol    Length  Info
40  0.000236664 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   PVA 102 32842 -> 5075 Client DESTROY_REQUEST(sid=117768961, ioid=268443651), 
42  0.042855197 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   PVA 115 32842 -> 5075 Client PUT(sid=117768961, ioid=268443652, sub=08), 
44  0.000454795 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   PVA 233 5075 -> 32842 Server PUT(ioid=268443652, sub=08), 
45  0.000553711 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   PVA 113 32842 -> 5075 Client PUT(sid=117768961, ioid=268443652, sub=00), 
46  0.000435356 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   PVA 100 5075 -> 32842 Server PUT(ioid=268443652, sub=00), 
47  0.000340852 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   PVA 102 32842 -> 5075 Client DESTROY_REQUEST(sid=117768961, ioid=268443652), 

Clearing the pva filter shows that this ~40ms interval ends with the (delayed) arrival of an ACK.

No. Time    Source  Destination Protocol    Length  Info
47  0.000340852 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   PVA 102 32842 -> 5075 Client DESTROY_REQUEST(sid=117768961, ioid=268443652), 
48  0.042193626 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   TCP 86  5075 → 32842 [ACK] Seq=868 Ack=435 Win=65536 Len=0 TSval=3005896866 TSecr=3005896824
49  0.000024206 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   PVA 115 32842 -> 5075 Client PUT(sid=117768961, ioid=268443653, sub=08), 

With TCP_NODELAY set, the PUT creation request is sent before any ACK deriving from the preceding DESTORY_REQUEST.

No. Time    Source  Destination Protocol    Length  Info
49  0.000033624 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   PVA 102 39420 -> 5075 Client DESTROY_REQUEST(sid=117768961, ioid=268443653), 
50  0.000075695 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   PVA 115 39420 -> 5075 Client PUT(sid=117768961, ioid=268443654, sub=08), 
51  0.000081687 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   TCP 86  5075 → 39420 [ACK] Seq=1029 Ack=536 Win=65536 Len=0 TSval=3005986106 TSecr=3005986106
52  0.000052146 fe80::9453:55ff:fe80:63a1   fe80::9453:55ff:fe80:63a1   PVA 233 5075 -> 39420 Server PUT(ioid=268443654, sub=08), 

[mdavidsaver]

Running this test code without setting TCP_NODELAY shows that the first iteration, with the extra overhead of channel creation, is unexpectedly faster than the following iterations.

#0 0.0134434 s
#1 0.043058 s
#2 0.0473097 s
#3 0.0437842 s
#4 0.0446515 s
#5 0.0432039 s
#6 0.0444871 s
#7 0.044105 s
#8 0.0439421 s
#9 0.044035 s

With TCP_NODELAY set, the first iteration time remains unchanged, while the subsequent iterations become much faster.

#0 0.0141374 s
#1 0.00181142 s
#2 0.00094667 s
#3 0.000754063 s
#4 0.000461921 s
#5 0.000373011 s
#6 0.000384887 s
#7 0.000283043 s
#8 0.000274032 s
#9 0.000268776 s

[rizzoa]

hi @mdavidsaver , I have installed in Windows 10 p4p 4.1.12, wiith pvxslibs 1.3.1. The python version is 3.11.8. When I do print the print(ctxt.get('pv1')), I almost always get this warning, 2024-02-21T21:07:37.915846888 WARN pvxs.client.io Unable to TCP_NODELAY: 10022 on 1276 It seems to be harmless, but it is little bit annoying, what I should do to remove it ? Please let me know if you need some more information. Thanks

[mdavidsaver]

@rizzoa Thank you for reporting. I see this warning in the PVXS unit tests as well.

2024-02-22T16:01:16.862248633 WARN pvxs.client.io Unable to TCP_NODELAY: 10022 on 600

It isn't not clear to me why this would be happening. I found one reference (https://github.com/redis/hiredis/issues/785#issuecomment-647156256) that points to a winsock specific race condition with async connect, which would apply to PVXS at present.

For a client, I guess the change to test would be moving this setsockopt() before connect().

Changing this would not be as simple as moving the setsockopt() up as with the call to bufferevent_socket_new(..., -1, ...) I am allowing the later bufferevent_socket_connect() to lazily create the socket. So some additional code would be needed to create the async. socket explicitly at the top of startConnecting().

https://github.com/mdavidsaver/pvxs/blob/2a56a085161c5c955b04c84965e02a9a8ec66355/src/clientconn.cpp#L65

...

https://github.com/mdavidsaver/pvxs/blob/2a56a085161c5c955b04c84965e02a9a8ec66355/src/clientconn.cpp#L72-L77

[mdavidsaver]

@rizzoa Please update and re-test. 4bd884719ef6e38cc5569b49ac22a8d9be0a833d avoids that warning in CI tests.

[rizzoa]

@mdavidsaver , thanks ! I have compiled with MSVC 2022 the commit 4bd8847 and now seems to works, no more annoying warnings !