Unexpected sequence.. - [TCP window full] in client's wireshark log

[arunkumarsit]

Hi mtcp team,

I Integrated mtcp with the user space NFS server. The NFS mount is working as expected. Thanks for all your support to achieve this!

Issue: I am facing following issue while copying files to NFS mount point. The copy command hangs and the following debug log is printed continuously by mtcp.

[ProcessTCPPayload: 619] Stream 5 data arrived. socket [15], len: 1448, ET: 0, IN: 1, OUT: 0
...
[ValidateSequence: 169] Window update request. (seq: 3120168939, rcv_wnd: 56)  --> **(Added by me)**
[ProcessTCPPacket:1226] ProcessTCPPacket() -> Stream 5: Unexpected sequence: 3120168939, expected: 3120168940

I am using 10G interface (82599ES) and SFP cable is connected peer-peer.

While checking the wireshark logs in client machine, I found that there are more TCP fragmentation happening as shown below.

The TCP window gets full in receiver end and there are no data transfer after that. The same setup works fine with Kernel TCP stack.

Do I need to do any configuration changes with mtcp? Kindly let me know whether I am missing anything. Following is my mtcp.conf file

# The underlying I/O module you want to use. Please
# enable only one out of the two.
io = dpdk

# Number of memory channels per processor socket (dpdk-only)
num_mem_ch = 4

num_cores = 1

# Used port (please adjust accordingly)
#------ DPDK ports -------#
port = dpdk0

# Maximum concurrency per core
max_concurrency = 10000

# Maximum number of socket buffers per core
# Set this to small value if there are many idle connections
max_num_buffers = 10000

# Receive buffer size of sockets
rcvbuf = 8192

# Send buffer size of sockets
sndbuf = 8192

# TCP timeout seconds
# (tcp_timeout = -1 can disable the timeout check)
tcp_timeout = 30

# TCP timewait seconds
tcp_timewait = 0

Please let me know your suggestions, Ideas. Thanks !

[arunkumarsit]

Hi,

On further analysis I found that there is a relation between TCP receive window size configured and the TCP segments received on server side.

The TCP receive window size is 8192 (8k) size. When I do transfer of file size less than 8k in parallel (100 files transferred), the copy works fine without any issues.

But when I copy a file with the size of 10k, the server receives the data as TCP segments from client. The segments are reaching server and the receive window size is keep on reducing (I can see it in Wireshark logs of reply from server to client). At one point there are no available TCP receive window. So the TCP ZeroWindow (len=0) is sent to client and the file transfer stopped.

So in my observation, I see there is a problem when the data size is greater than the Receive window size of mTCP where TCP segmentation happens from client side. And the server with mTCP stack is not clearing the window used by the segmented packets.

Kindly let me know if you know the reason for this behavior.

BTW, I have one more setup with the same HW and the application running on it. In that setup, the client is not sending the segmented packets. And the file copy is working fine irrespective of Window size (8k) compared to the file size we are copying.

[ajamshed]

Hi Arun,

Can you please check if changing the rcvbuf/sndbuf parameters mentioned in the mtcp.conf file improve the performance? These parameters are statically assigned during initialization and may indirectly effect your NFS server.

[azamikram]

@arunkumarsit were you able to solve your issue?

I'm facing a similar problem with my setup. I have a system where the receiver reads the responses a bit late from the point when they had arrived at the receiver side (it is intentional). This means my receiver buffer becomes full and it had to send ZeroWindow which blocks the client from sending any further messages. However, the client never starts again, even after I have read everything on the receiver side.

The point where the receiver buffer becomes full can be delayed by increasing the size of the receiver buffer. But even then a point eventually comes when the receiver has to send ZeroWindow and then everything stops.