search

multipath-tcp / mptcp

⚠️⚠️⚠️ Deprecated 🚫 Out-of-tree Linux Kernel implementation of MultiPath TCP. 👉 Use https://github.com/multipath-tcp/mptcp_net-next repo instead ⚠️⚠️⚠️
https://github.com/multipath-tcp/mptcp_net-next
Other
890 stars 335 forks source link

Redundant scheduler behaviour #271

Open sujithrengan opened 6 years ago

sujithrengan commented 6 years ago

Hi,

I have been testing MPTCP redundant scheduler.

Test setup:

Behaviour:

Initial data packets that are less than 1 MSS are scheduled in both the interfaces. (58fa7ad patch is applied in the device) When uploading the file ~10kB, the data packets are not redundantly scheduled, but behaves like default scheduler. When trying the same upload with a larger file ~1mB, initial 15-20 packets are not redundantly sent, but later packets are redundantly sent. Also, after the upload, final data packets that are less than 1 MSS are again redundantly scheduled in both the interfaces.

I'm not sure why the data packets during upload is not redundantly sent always. Any help would be much appreciated. Thanks in advance.

cpaasch commented 6 years ago

Can you share a packet-trace? Thanks!

Also, please provide details as to which version of MPTCP is installed and what the configuration is.

sujithrengan commented 6 years ago

Hi,

Sorry to get back late. Did some further analysis.

Below is the tcpdump logs for a session (Redacted server IP to 1.2.3.4):

`19:02:21.512732 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [S], seq 625133855, win 65535, options [mss 1460,sackOK,TS val 287924 ecr 0,nop,wscale 8,mptcp capable {0xf989fd906f83c00a}], length 0
19:02:21.966743 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [S.], seq 1873996102, ack 625133856, win 26787, options [mss 1300,sackOK,TS val 3906128375 ecr 287924,nop,wscale 7,mptcp capable {0x485366e2534ff157}], length 0
19:02:21.968351 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], ack 1, win 343, options [nop,nop,TS val 287970 ecr 3906128375,mptcp capable {0xf989fd906f83c00a,0x485366e2534ff157},mptcp dss ack 1502162830], length 0
19:02:21.968964 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], ack 1, win 343, options [nop,nop,TS val 287970 ecr 3906128375,mptcp add-addr id 8 100.65.182.64,mptcp dss ack 1502162830], length 0
19:02:21.979183 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [P.], seq 1:340, ack 1, win 343, options [nop,nop,TS val 287971 ecr 3906128375,mptcp dss ack 1502162830 seq 1426344276 subseq 1 len 339,nop,nop], length 339
19:02:22.483149 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [.], ack 340, win 218, options [nop,nop,TS val 3906128584 ecr 287971,mptcp dss ack 1426344615], length 0
19:02:22.483917 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [P.], seq 1:146, ack 340, win 218, options [nop,nop,TS val 3906128584 ecr 287971,mptcp dss ack 1426344615 seq 1502162830 subseq 1 len 145,nop,nop], length 145
19:02:22.486769 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [S], seq 2727580873, win 65535, options [mss 1390,sackOK,TS val 288022 ecr 0,nop,wscale 8,mptcp join id 8 token 0x2ec20a8f nonce 0x89f3affe], length 0
19:02:22.487673 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], ack 146, win 344, options [nop,nop,TS val 288022 ecr 3906128584,mptcp dss ack 1502162975], length 0
19:02:22.490467 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [P.], seq 340:440, ack 146, win 344, options [nop,nop,TS val 288022 ecr 3906128584,mptcp dss ack 1502162975 seq 1426344615 subseq 340 len 100,nop,nop], length 100
19:02:22.845110 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [.], ack 440, win 218, options [nop,nop,TS val 3906129157 ecr 288022,mptcp dss ack 1426344715], length 0
19:02:23.269968 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [S.], seq 183743103, ack 2727580874, win 26787, options [mss 1300,sackOK,TS val 1354216036 ecr 288022,nop,wscale 7,mptcp join id 2 hmac 0x602e315a9a04f1d0 nonce 0x6571c308], length 0
19:02:23.271316 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], ack 1, win 670, options [nop,nop,TS val 288100 ecr 1354216036,mptcp join hmac 0xc81be502e341e18d5c0c16181d8b9b61397a7c13], length 0
19:02:24.045365 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 1, win 427, options [nop,nop,TS val 1354216798 ecr 288100,mptcp dss ack 1426344715], length 0
19:02:24.047030 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [P.], seq 1:87, ack 1, win 670, options [nop,nop,TS val 288178 ecr 1354216036,mptcp dss ack 1502162975 seq 1426344715 subseq 1 len 86,nop,nop], length 86
19:02:24.901846 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 87, win 427, options [nop,nop,TS val 1354217662 ecr 288178,mptcp dss ack 1426344801], length 0
19:02:24.927817 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [P.], seq 1:48, ack 87, win 427, options [nop,nop,TS val 1354217664 ecr 288178,mptcp dss ack 1426344801 seq 1502162975 subseq 1 len 47,nop,nop], length 47
19:02:24.929316 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], ack 48, win 670, options [nop,nop,TS val 288266 ecr 1354217664,mptcp dss ack 1502163022], length 0
19:02:24.933451 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [P.], seq 440:750, ack 146, win 670, options [nop,nop,TS val 288266 ecr 3906129157,mptcp dss ack 1502163022 seq 1426344801 subseq 440 len 310,nop,nop], length 310
19:02:24.938234 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [P.], seq 87:397, ack 48, win 670, options [nop,nop,TS val 288267 ecr 1354217664,mptcp dss ack 1502163022 seq 1426344801 subseq 87 len 310,nop,nop], length 310
19:02:24.946021 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [P.], seq 146:193, ack 440, win 427, options [nop,nop,TS val 3906131278 ecr 288022,mptcp dss ack 1426344801 seq 1502162975 subseq 146 len 47,nop,nop], length 47
19:02:24.985892 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], ack 193, win 670, options [nop,nop,TS val 288272 ecr 3906131278,mptcp dss ack 1502163022], length 0
19:02:25.114224 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [.], ack 750, win 436, options [nop,nop,TS val 3906131546 ecr 288266,mptcp dss ack 1426345111], length 0
19:02:25.118261 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [P.], seq 193:456, ack 750, win 436, options [nop,nop,TS val 3906131551 ecr 288266,mptcp dss ack 1426345111 seq 1502163022 subseq 193 len 263,nop,nop], length 263
19:02:25.118563 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], ack 456, win 672, options [nop,nop,TS val 288285 ecr 3906131551,mptcp dss ack 1502163285], length 0
19:02:25.123009 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [P.], seq 750:825, ack 456, win 672, options [nop,nop,TS val 288285 ecr 3906131551,mptcp dss ack 1502163285 seq 1426345111 subseq 750 len 75,nop,nop], length 75
19:02:25.124151 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [P.], seq 397:472, ack 48, win 672, options [nop,nop,TS val 288285 ecr 1354217664,mptcp dss ack 1502163285 seq 1426345111 subseq 397 len 75,nop,nop], length 75
19:02:25.248316 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [P.], seq 48:311, ack 87, win 436, options [nop,nop,TS val 1354217937 ecr 288178,mptcp dss ack 1426345111 seq 1502163022 subseq 48 len 263,nop,nop], length 263
19:02:25.286157 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], ack 311, win 674, options [nop,nop,TS val 288302 ecr 1354217937,mptcp dss ack 1502163285], length 0
19:02:25.292485 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [P.], seq 456:494, ack 825, win 436, options [nop,nop,TS val 3906131723 ecr 288285,mptcp dss ack 1426345186 seq 1502163285 subseq 456 len 38,nop,nop], length 38
19:02:25.327217 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], ack 494, win 674, options [nop,nop,TS val 288306 ecr 3906131723,mptcp dss ack 1502163323], length 0
19:02:25.357130 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [P.], seq 311:349, ack 87, win 436, options [nop,nop,TS val 1354218109 ecr 288178,mptcp dss ack 1426345186 seq 1502163285 subseq 311 len 38,nop,nop], length 38
19:02:25.357925 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], ack 349, win 674, options [nop,nop,TS val 288309 ecr 1354218109,mptcp dss ack 1502163323], length 0
19:02:26.987218 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [P.], seq 397:472, ack 349, win 674, options [nop,nop,TS val 288472 ecr 1354218109,mptcp dss ack 1502163323 seq 1426345111 subseq 397 len 75,nop,nop], length 75
19:02:27.940229 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 397, win 444, options [nop,nop,TS val 1354220699 ecr 288266,mptcp dss ack 1426345186], length 0
19:02:28.594672 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 472, win 444, options [nop,nop,TS val 1354221348 ecr 288266,mptcp dss ack 1426345186], length 0
19:02:29.270293 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [P.], seq 825:855, ack 494, win 674, options [nop,nop,TS val 288700 ecr 3906131723,mptcp dss ack 1502163323 seq 1426345186 subseq 825 len 30,nop,nop], length 30
19:02:29.274621 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [P.], seq 472:502, ack 349, win 674, options [nop,nop,TS val 288700 ecr 1354221348,mptcp dss ack 1502163323 seq 1426345186 subseq 472 len 30,nop,nop], length 30
19:02:29.275841 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], seq 502:1770, ack 349, win 674, options [nop,nop,TS val 288701 ecr 1354221348,mptcp dss ack 1502163323 seq 1426345216 subseq 502 len 1268,nop,nop], length 1268
19:02:29.296359 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], seq 855:2123, ack 494, win 674, options [nop,nop,TS val 288703 ecr 3906131723,mptcp dss ack 1502163323 seq 1426346484 subseq 855 len 1268,nop,nop], length 1268
19:02:29.302353 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], seq 1770:3038, ack 349, win 674, options [nop,nop,TS val 288703 ecr 1354221348,mptcp dss ack 1502163323 seq 1426347752 subseq 1770 len 1268,nop,nop], length 1268
19:02:29.307589 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 472, win 444, options [nop,nop,TS val 1354222074 ecr 288472,nop,nop,sack 1 {397:472},mptcp dss ack 1426345186], length 0
19:02:29.314934 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], seq 2123:3391, ack 494, win 674, options [nop,nop,TS val 288704 ecr 3906131723,mptcp dss ack 1502163323 seq 1426349020 subseq 2123 len 1268,nop,nop], length 1268
19:02:29.320943 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], seq 3038:4306, ack 349, win 674, options [nop,nop,TS val 288705 ecr 1354222074,mptcp dss ack 1502163323 seq 1426350288 subseq 3038 len 1268,nop,nop], length 1268
19:02:29.333397 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], seq 3391:4659, ack 494, win 674, options [nop,nop,TS val 288706 ecr 3906131723,mptcp dss ack 1502163323 seq 1426351556 subseq 3391 len 1268,nop,nop], length 1268
19:02:29.340008 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], seq 4306:5574, ack 349, win 674, options [nop,nop,TS val 288707 ecr 1354222074,mptcp dss ack 1502163323 seq 1426352824 subseq 4306 len 1268,nop,nop], length 1268
19:02:29.352046 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], seq 4659:5927, ack 494, win 674, options [nop,nop,TS val 288708 ecr 3906131723,mptcp dss ack 1502163323 seq 1426354092 subseq 4659 len 1268,nop,nop], length 1268
19:02:29.358491 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], seq 5574:6842, ack 349, win 674, options [nop,nop,TS val 288709 ecr 1354222074,mptcp dss ack 1502163323 seq 1426355360 subseq 5574 len 1268,nop,nop], length 1268
19:02:29.370657 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], seq 5927:7195, ack 494, win 674, options [nop,nop,TS val 288710 ecr 3906131723,mptcp dss ack 1502163323 seq 1426356628 subseq 5927 len 1268,nop,nop], length 1268
19:02:29.493059 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [.], ack 2123, win 464, options [nop,nop,TS val 3906135925 ecr 288700,mptcp dss ack 1426345216], length 0
19:02:29.507954 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [.], ack 4659, win 504, options [nop,nop,TS val 3906135937 ecr 288704,mptcp dss ack 1426345216], length 0
19:02:29.552430 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [.], ack 7195, win 543, options [nop,nop,TS val 3906135983 ecr 288708,mptcp dss ack 1426345216], length 0
19:02:30.573690 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 472, win 444, options [nop,nop,TS val 1354222074 ecr 288472,nop,nop,sack 1 {397:472},mptcp dss ack 1426345186], length 0
19:02:30.699332 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 502, win 543, options [nop,nop,TS val 1354223520 ecr 288700,mptcp dss ack 1426345216], length 0
19:02:30.699361 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 1770, win 563, options [nop,nop,TS val 1354223520 ecr 288701,mptcp dss ack 1426347752], length 0
19:02:30.709010 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 3038, win 583, options [nop,nop,TS val 1354223530 ecr 288703,mptcp dss ack 1426350288], length 0
19:02:30.709038 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 4306, win 603, options [nop,nop,TS val 1354223530 ecr 288705,mptcp dss ack 1426352824], length 0
19:02:30.709058 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 5574, win 622, options [nop,nop,TS val 1354223530 ecr 288707,mptcp dss ack 1426355360], length 0
19:02:30.709103 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 6842, win 642, options [nop,nop,TS val 1354223530 ecr 288709,mptcp dss ack 1426357896], length 0
19:02:38.313014 IP 192.168.43.137.51204 > 74.125.200.188.5228: Flags [P.], seq 238180083:238180118, ack 2167231485, win 347, options [nop,nop,TS val 289604 ecr 4071569539], length 35
19:02:38.548934 IP 74.125.200.188.5228 > 192.168.43.137.51204: Flags [P.], seq 1:34, ack 35, win 244, options [nop,nop,TS val 4071679816 ecr 289604], length 33
19:02:38.550044 IP 192.168.43.137.51204 > 74.125.200.188.5228: Flags [.], ack 34, win 347, options [nop,nop,TS val 289628 ecr 4071679816], length 0
19:02:42.152591 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [P.], seq 494:704, ack 7195, win 642, options [nop,nop,TS val 3906148154 ecr 288708,mptcp dss ack 1426357896 seq 1502163323 subseq 494 len 210,nop,nop], length 210
19:02:42.153482 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], ack 704, win 675, options [nop,nop,TS val 289988 ecr 3906148154,mptcp dss ack 1502163533], length 0
19:02:42.969838 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [P.], seq 349:559, ack 6842, win 642, options [nop,nop,TS val 1354234540 ecr 288709,mptcp dss ack 1426357896 seq 1502163323 subseq 349 len 210,nop,nop], length 210
19:02:42.971048 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], ack 559, win 677, options [nop,nop,TS val 290070 ecr 1354234540,mptcp dss ack 1502163533], length 0
19:02:43.699736 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [P.], seq 349:559, ack 6842, win 642, options [nop,nop,TS val 1354236407 ecr 288709,mptcp dss ack 1426357896 seq 1502163323 subseq 349 len 210,nop,nop], length 210
19:02:43.701451 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], ack 559, win 677, options [nop,nop,TS val 290143 ecr 1354236407,nop,nop,sack 1 {349:559},mptcp dss ack 1502163533], length 0
19:02:56.826626 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [P.], seq 6842:7538, ack 559, win 677, options [nop,nop,TS val 291456 ecr 1354236407,mptcp dss ack 1502163533 seq 1426357896 subseq 6842 len 696,nop,nop], length 696
19:02:59.627061 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [P.], seq 6842:7538, ack 559, win 677, options [nop,nop,TS val 291736 ecr 1354236407,mptcp dss ack 1502163533 seq 1426357896 subseq 6842 len 696,nop,nop], length 696
19:03:00.435942 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 7538, win 653, options [nop,nop,TS val 1354253258 ecr 291456,mptcp dss ack 1426358592], length 0
19:03:00.435969 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 7538, win 653, options [nop,nop,TS val 1354253258 ecr 291736,nop,nop,sack 1 {6842:7538},mptcp dss ack 1426358592], length 0
19:03:00.440703 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [P.], seq 559:607, ack 7538, win 653, options [nop,nop,TS val 1354253262 ecr 291736,mptcp dss ack 1426358592 seq 1502163533 subseq 559 len 48,nop,nop], length 48
19:03:00.441281 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [.], ack 607, win 677, options [nop,nop,TS val 291817 ecr 1354253262,mptcp dss ack 1502163581], length 0
19:03:00.458657 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [P.], seq 7195:7225, ack 704, win 677, options [nop,nop,TS val 291818 ecr 3906148154,mptcp dss ack 1502163581 seq 1426358592 subseq 7195 len 30,nop,nop], length 30
19:03:00.461577 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [P.], seq 7538:7568, ack 607, win 677, options [nop,nop,TS val 291819 ecr 1354253262,mptcp dss ack 1502163581 seq 1426358592 subseq 7538 len 30,nop,nop], length 30
19:03:00.676798 IP 1.2.3.4.443 > 100.65.182.64.42365: Flags [.], ack 7568, win 653, options [nop,nop,TS val 1354253500 ecr 291819,mptcp dss ack 1426358622], length 0
19:03:00.678091 IP 100.65.182.64.42365 > 1.2.3.4.443: Flags [P.], seq 7568:7625, ack 607, win 677, options [nop,nop,TS val 291841 ecr 1354253262,mptcp dss ack 1502163581 seq 1426358622 subseq 7568 len 57,nop,nop], length 57
19:03:00.780263 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [P.], seq 704:752, ack 7195, win 653, options [nop,nop,TS val 3906166876 ecr 289988,mptcp dss ack 1426358592 seq 1502163533 subseq 704 len 48,nop,nop], length 48
19:03:00.780817 IP 192.168.43.137.55310 > 1.2.3.4.443: Flags [.], ack 752, win 677, options [nop,nop,TS val 291851 ecr 3906166876,mptcp dss ack 1502163581], length 0
19:03:00.850854 IP 1.2.3.4.443 > 192.168.43.137.55310: Flags [P.], seq 752:800, ack 7225, win 653, options [nop,nop,TS val 3906167284 ecr 291818,mptcp dss ack 1426358679 seq 1502163581 subs`

Configs:

MPTCP version installed on the device is 0.93
mptcp_scheduler=redundant 
mptcp_path_manager=fullmesh
mptcp_checksum=0
mptcp_syn_retries=3

Further analysis: Did some further testing, looks like the issue is related to nagle test failing on one subflow, Later the same packet is sent on the next_subflow but not retried on the original subflow. The comment from redundant_next_skb_from_queue explains the same:

  1. sk_data->skb was already sent on another subflow: with regard to the redundant semantic, we have to chedule it again. However, we keep it simple and ignore it, as it was already sent by another subflow. This might be changed in the future.

So can you confirm this is the expected behaviour acc. to current implementation of the scheduler Thanks in advance.

cpaasch commented 6 years ago

Yeah, looking more at this code, it seems like when the send-buffer is full, we are not sending the data in a redundant way.

I will reach out to the author of this module to see if he has cycles to work on it.

AlexanderFroemmgen commented 6 years ago

I will try to have a look next weekend.

AlexanderFroemmgen commented 5 years ago

Looking at the provided trace, it really looks a bit fishy. I annotated the trace a bit:

Client 2 Server

19:02:21.979183 IP 192.168.43.137.55310 > 1.2.3.4.443     seq 1426344276 len 339    // Second subflow not established yet.
19:02:22.490467 IP 192.168.43.137.55310 > 1.2.3.4.443.    seq 1426344615 len 100    // Second subflow not established yet (Ongoing handshake).

19:02:24.047030 IP 100.65.182.64.42365 > 1.2.3.4.443.     seq 1426344715 len 86     // Problem here?

19:02:24.933451 IP 192.168.43.137.55310 > 1.2.3.4.443:    seq 1426344801 len 310    // The first example of Client 2 Server redundancy -> Client side is configured to use redundant scheduler
19:02:24.938234 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426344801 len 310

19:02:25.123009 IP 192.168.43.137.55310 > 1.2.3.4.443:    seq 1426345111 len 75 // Second example, this looks as expected :-)
19:02:25.124151 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426345111 len 75
19:02:26.987218 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426345111 len 75 // Seems to be a retransmission…

19:02:29.270293 IP 192.168.43.137.55310 > 1.2.3.4.443:    seq 1426345186 len 30 // Still good (all with Push Flag so far)
19:02:29.274621 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426345186 len 30

19:02:29.275841 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426345216 len 1268   // Here, the problem starts. the packet size is a bit unexpected…
19:02:29.296359 IP 192.168.43.137.55310 > 1.2.3.4.443:    seq 1426346484 len 1268   // Looks more like a round-robin scheduling :-(
19:02:29.302353 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426347752 len 1268
19:02:29.314934 IP 192.168.43.137.55310 > 1.2.3.4.443:    seq 1426349020 len 1268
19:02:29.320943 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426350288 len 1268
19:02:29.333397 IP 192.168.43.137.55310 > 1.2.3.4.443:    seq 1426351556 len 1268
19:02:29.340008 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426352824 len 1268
19:02:29.352046 IP 192.168.43.137.55310 > 1.2.3.4.443:    seq 1426354092 len 1268
19:02:29.358491 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426355360 len 1268
19:02:29.370657 IP 192.168.43.137.55310 > 1.2.3.4.443:    seq 1426356628 len 1268

19:02:56.826626 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426357896 len 696
19:02:59.627061 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426357896 len 696    // Timeout and retransmission?

19:03:00.458657 IP 192.168.43.137.55310 > 1.2.3.4.443:    seq 1426358592 len 30     // Looks good again
19:03:00.461577 IP 100.65.182.64.42365 > 1.2.3.4.443:     seq 1426358592 len 30

Server 2 Client

19:02:22.483917 IP 1.2.3.4.443 > 192.168.43.137.55310     seq 1502162830 len 145            // Second subflow not established yet.

19:02:24.927817 IP 1.2.3.4.443 > 100.65.182.64.42365      seq 1502162975 len 47
19:02:24.946021 IP 1.2.3.4.443 > 192.168.43.137.55310.    seq 1502162975 len 47 // 20ms time difference between both packets, but redundant. Assuming that the trace was captured on the client side, the time difference is caused by the network.

19:02:25.118261 IP 1.2.3.4.443 > 192.168.43.137.55310:    seq 1502163022 len 263        // Looks good
19:02:25.248316 IP 1.2.3.4.443 > 100.65.182.64.42365:     seq 1502163022 len 263

19:02:25.292485 IP 1.2.3.4.443 > 192.168.43.137.55310:    seq 1502163285 len 38     // Looks good
19:02:25.357130 IP 1.2.3.4.443 > 100.65.182.64.42365:     seq 1502163285 len 38 

19:02:42.152591 IP 1.2.3.4.443 > 192.168.43.137.55310:    seq 1502163323 len 210        // looks good
19:02:42.969838 IP 1.2.3.4.443 > 100.65.182.64.42365:     seq 1502163323 len 210
19:02:43.699736 IP 1.2.3.4.443 > 100.65.182.64.42365:     seq 1502163323 len 210    // Retransmission

19:03:00.440703 IP 1.2.3.4.443 > 100.65.182.64.42365:     seq 1502163533 len 48     // looks good
19:03:00.780263 IP 1.2.3.4.443 > 192.168.43.137.55310:    seq 1502163533 len 48

I will try to reproduce your setup and update this thread in a few days.

AlexanderFroemmgen commented 5 years ago

I just tested a basic setup with the recent mptcp release, two subflows, iperf, and the redundant scheduler. There, it works as expected, i.e., both subflows are saturated but the overall throughput is the throughput of the best subflow. Thus, there are at least a few configurations which are working ;-)

@sujithrengan : Can you share some more details about your testsetup, e.g., i) can you share your sample application, ii) which Nagle configuration you are using, iii) if there is any particular reason for the 1268 packet size. Did you try other applications, e.g., iPerf or SSH?