MPTCP connection becomes stale

mario-minati commented 2 days ago

Expected Behavior

We would expect that both OMR sides could run for a long time without problems.

Current Behavior

We discovered problems with the VPS part of OMR after running for about 14 days.

The problem occured as a massiv speed degradation and failure of most reuqest which were routed to the VPS, Those packages which were bypassed through OMR-bypass settings had no problems.

The graphs of our VPS look strange: grafik

There are no problems with free disk space, but there are hundreds and hundreds processes like this:

root     1952539  0.0  0.0   2596   108 ?        S    May22   0:00  \_ /bin/sh -c ss -M | grep 195.50.149.xxx
root     1952540  0.0  0.0   6632   612 ?        S    May22   0:00  |   \_ ss -M
root     1952541  0.0  0.0   6444   144 ?        S    May22   0:00  |   \_ grep 195.50.149.xxx

Every day has around 20 processes like this. 195.50.149.xxx is the fixed ip of our master internet connection. This includes every single day since the last reboot.

If helpfull we can post the complete output of ps aufx.

After a reboot everything was working smoothly again.

Possible Solution

Cleanup of old sh processes?

Steps to Reproduce the Problem

Run OMR VPS for multipe days.

Context (Environment)

Our network overview: grafik

Specifications

OpenMPTCProuter version: openmptcprouter v0.60-6.1 r0+24843-acf40c022e
OpenMPTCProuter VPS version: 0.1030
OpenMPTCProuter VPS provider: Hetzner
OpenMPTCProuter platform: x86_64
Country: Germany

Ysurac commented 2 days ago

Strange, I don't have this issue on my servers. What is the result of ss -M | grep 195.50.149.xxx if you run it manually ? I added a timeout as a workaround solution for now in snapshot script ( https://github.com/Ysurac/openmptcprouter/wiki/Snapshots )

mario-minati commented 2 days ago

The output doesn't show any obvious problems: https://pastebin.com/zj718dZh

Thanks again for the quick reply.

mario-minati commented 2 days ago

I just saw that these shells are created by the omr-admin script:

root         827  0.1  4.5 195768 90108 ?        Ssl  Jun27   3:50 python3 /usr/local/bin/omr-admin.py --host=::
root         828  0.2  7.9 333972 156572 ?       Ssl  Jun27   6:33 python3 /usr/local/bin/omr-admin.py
root       80479  0.0  0.0   2596   932 ?        S    Jun27   0:00  \_ /bin/sh -c ss -M | grep 195.50.149.xxx
root       80481  0.0  0.1   6444  2024 ?        S    Jun27   0:00  |   \_ grep 195.50.149.xx
root      295841  0.0  0.0   2596   944 ?        S    Jun27   0:00  \_ /bin/sh -c ss -M | grep 195.50.149.xxx
root      295842  0.0  0.1   6632  3424 ?        S    Jun27   0:00  |   \_ ss -M
root      295843  0.0  0.1   6444  2008 ?        S    Jun27   0:00  |   \_ grep 195.50.149.xxx

Maybe this is related to having the OMR web admin always open in one tab of my browser and showing the status page which is always refreshing.

Ysurac / openmptcprouter