search

multipath-tcp / mptcp

⚠️⚠️⚠️ Deprecated 🚫 Out-of-tree Linux Kernel implementation of MultiPath TCP. 👉 Use https://github.com/multipath-tcp/mptcp_net-next repo instead ⚠️⚠️⚠️
https://github.com/multipath-tcp/mptcp_net-next
Other
887 stars 334 forks source link

Many __mptcp_init4_subsockets mptcp_add_sock failed ret: -1 kernel errors lately. #516

Open ioogithub opened 9 months ago

ioogithub commented 9 months ago

Hi,

I am using MPTCP in the openmptcprouter version v0.59.1-5.4 r0+16594-ce92de8c8c implementation. Linux OpenMPTCProuter 5.4.194 #0 SMP Tue May 17 22:11:28 2022 x86_64 GNU/Linux

I have been getting a log of kernel errors in the system log in the past few days:

Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.141479] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.141940] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.142388] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.142822] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.143450] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.143692] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.143877] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.144350] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.144755] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.145150] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:30 2023 kern.warn kernel: [ 6649.499339] net_ratelimit: 5 callbacks suppressed
Mon Oct 16 15:55:30 2023 kern.err kernel: [ 6649.499341] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:30 2023 kern.err kernel: [ 6649.500079] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:30 2023 kern.err kernel: [ 6649.500423] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:05:16 2023 kern.err kernel: [ 7235.704584] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:05:16 2023 kern.err kernel: [ 7235.704930] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.496316] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.496645] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.496950] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.497257] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.497546] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.497812] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.498066] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.498318] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.498548] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.499893] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:13:10 2023 kern.warn kernel: [ 7709.300992] net_ratelimit: 24 callbacks suppressed
Mon Oct 16 16:13:10 2023 kern.err kernel: [ 7709.300993] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:13:10 2023 kern.err kernel: [ 7709.301439] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:13:10 2023 kern.err kernel: [ 7709.301627] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.505203] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.505398] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.505588] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.505801] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.505984] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.506165] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.506406] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.506592] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.506779] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.506963] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:38:46 2023 kern.warn kernel: [ 9245.244057] net_ratelimit: 18 callbacks suppressed
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.710471] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.710661] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.710856] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.711040] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.711225] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.711404] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.711588] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:39 2023 kern.err kernel: [ 9598.505907] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:39 2023 kern.err kernel: [ 9598.506096] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:39 2023 kern.err kernel: [ 9598.506274] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:50:32 2023 kern.err kernel: [ 9951.776239] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:50:32 2023 kern.err kernel: [ 9951.776436] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:50:32 2023 kern.err kernel: [ 9951.776617] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:50:32 2023 kern.err kernel: [ 9951.776787] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:50:32 2023 kern.err kernel: [ 9951.776952] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.839483] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.839668] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.841189] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.841363] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.841524] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.841736] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.841920] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.842077] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.842235] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.842392] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:26 2023 kern.warn kernel: [10845.189114] net_ratelimit: 6 callbacks suppressed
Mon Oct 16 17:05:26 2023 kern.err kernel: [10845.189124] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:26 2023 kern.err kernel: [10845.189499] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:26 2023 kern.err kernel: [10845.189658] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076000] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076173] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076333] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076490] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076653] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076826] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076996] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.077149] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.077309] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.077469] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:07:26 2023 kern.err kernel: [10965.502737] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:07:26 2023 kern.err kernel: [10965.505369] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:07:26 2023 kern.err kernel: [10965.505538] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:07:26 2023 kern.err kernel: [10965.505695] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:07:26 2023 kern.err kernel: [10965.505847] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:08:27 2023 kern.err kernel: [11026.951586] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:08:27 2023 kern.err kernel: [11026.951751] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:08:27 2023 kern.err kernel: [11026.951901] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:08:27 2023 kern.err kernel: [11026.952047] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.729882] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.730051] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.730668] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.730831] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.730983] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.731170] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.731314] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.731453] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.731576] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.731700] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:26:38 2023 kern.err kernel: [12117.466211] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:26:38 2023 kern.err kernel: [12117.466530] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:26:38 2023 kern.err kernel: [12117.466689] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:26:38 2023 kern.err kernel: [12117.466855] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:27:11 2023 kern.err kernel: [12150.735869] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1

I don't see any other errors in the log that would indicate issues.

Here are some statistics:

MPCapableSYNTX                      12194
MPCapableSYNACKRX                   11464
MPCapableACKRX                      4
MPCapableFallbackACK                0
MPCapableFallbackSYNACK             700
MPCapableRetransFallback            0
MPTCPCsumEnabled                    0
MPTCPRetrans                        4479
MPFailRX                            0
MPCsumFail                          0
MPFastcloseRX                       931
MPFastcloseTX                       2592
MPFallbackAckSub                    0
MPFallbackAckInit                   0
MPFallbackDataSub                   0
MPFallbackDataInit                  0
MPRemoveAddrSubDelete               858
MPJoinNoTokenFound                  0
MPJoinAlreadyFallenback             0
MPJoinSynTx                         1268212
MPJoinSynRx                         12
MPJoinSynAckRx                      238691
MPJoinSynAckHMacFailure             0
MPJoinAckRx                         12
MPJoinAckHMacFailure                0
MPJoinAckMissing                    0
MPJoinAckRTO                        1
MPJoinAckRexmit                     2560
NoDSSInWindow                       0
DSSNotMatching                      0
InfiniteMapRx                       0
DSSNoMatchTCP                       0
DSSTrimHead                         0
DSSSplitTail                        0
DSSPurgeOldSubSegs                  0
AddAddrRx                           11718
AddAddrTx                           11464
RemAddrRx                           858
RemAddrTx                           0
MPJoinAlternatePort                 0
MPCurrEstab                         116

I saw in this issue: https://github.com/multipath-tcp/mptcp/issues/245 that you support 32 subflows per connection. I am aggregating two wan connections on the client (openmptcprouter) each with 1 IP address and 1 IP on the server (VPS).

I have Fullmesh subflows for each pair of IP addresses set to a value of 3. What do these errors mean, why is the add_sock process failing and when it does what are the implications, does this mean these connections are not aggregating properly. How can I troubleshoot or fix the problem?

Thank you.

matttbe commented 9 months ago

Hello,

Linux OpenMPTCProuter 5.4.194 #0 SMP Tue May 17 22:11:28 2022 x86_64 GNU/Linux

This version is a bit old, maybe the issue has already been fixed.

If you have mptcp_add_sock failed ret: -1, it means No more space for more subflows. So it looks like you are trying to create too many subflows.

I have Fullmesh subflows for each pair of IP addresses set to a value of 3.

I'm not sure what it represents (best is to report this issue to OpenMPTCProuter devs I think) but I guess it will try to create more subflows than needed.