Does mptcp support epoll

[alex1230608]

Does mptcp with multiple subflows support epoll socket programming and non-blocking socket?

I am using mptcp v0.95 compiled on CentOS 7.6. I have tried my program with several different setups:

1. linux tcp + cubic: works fine
2. mptcp 1 subflow + cubic: similar to 1
3. mptcp 2 subflows + cubic: traffic can be captured by tcpdump but application cannot receive any data.
4. mptcp 2 subflows + lia: throughput is better than 1 and 2.

At first glance, multiple subflows seems not working with cubic, but when I use iperf3 with the same setup, the data can be received. Does this mean epoll is not well supported by mptcp?

[cpaasch]

epoll is supported correctly by MPTCP. Can you give us more details as to what app you are using, and share a tcpdump as well?

[alex1230608]

The attached is the tcpdump. The program is very long since it combines other features I need, and to abstract the related code snippet requires some time. Therefore, allow me to attach the tcpdump result captured from the sender first.

The setup is mptcp 4 subflows + cubic.

Please ignore this part,

the delay is caused by my program on purpose

and I just forgot.

I just noticed that the problem seems that most of the data have not been transmitted and the last subflow has not been established until 7 seconds after the first SYN packet. #############################################

mptcp_cubic_4subflows_sender.txt

[cpaasch]

In this trace I don't see anything that shows that the app is not able to read. The data-ack is making progress and the window go down to 0.

Are you sure that epoll is not waking up the app?

[alex1230608]

It did wake up the app, but it didn't wake it up until nearly the end of the transmission. What I observed was

1. In the beginning of the transmission (18:23:30.139764 in the tcpdump), the sender received an application ack from the receiver saying 43KB have been received.
2. 139ms later, sender received an application ack saying 3.34MB have been received.
3. 6.2 sec later, sender received the final application ack saying that 6GB (all the data) have been received.

It seems that the mptcp will not trigger the epoll until a huge amount of data have been received. As my another post mentioned, the CPU utilization is 100% on both sides during the transmission. Can I conclude that this is because CPU is too busy when handling the transmission and therefore postpone the trigger to epoll?

[cpaasch]

It can't be that the receiver only wakes up after 6GB have been received. The window is much smaller than that and it is guaranteed that the wakeup happens before. I guess it is more of a problem in your app that I guess is reading as much as it can from the socket and only when it gets a EWOULDBLOCK will stop reading and finally sends the application-level ACK. Is that right?

[alex1230608]

I see. I am doing exactly what you are saying here. We will fix that part. Thank you!! However, just out of curious, why linux tcp does not have this problem?

[cpaasch]

I guess it is because you are CPU-limited here on the application's side.

[cpaasch]

Closing.