search

xcode75 / XManagerPlus

An Xray Manager
https://xcode75.github.io/XMPlusDocs/
265 stars 49 forks source link

constantly lag on every min #178

Closed letmefind closed 1 year ago

letmefind commented 1 year ago

There is some issue that makes causes lagging every a min and takes about 10-15ish sec on all servers regardless of protocol types. during the lag CPU consumption exceed over 300-500%.

Troubleshooting: In order to spot potential issue Three servers have been deployed as follow for a few user groups : Two dedicated server(6CPU/8G RAM/1GB Network)

1- Server A (Trojan+TLS/GPRC+Nginx)
2- Server B (VLESS +TLS/GPRC+Nginx)

A server (2CPU/4G RAM/1Gb Network)

3- Server C (VMESS +TCP+TLS)

XMPLUS Panel running on private server separately.

Noticed during the lag some users lose VPN connectivity or have no internet and it would be back to normal in 10-15 sec. No time correlation was found between lag time and API update periods span-time (120s) as well. here are a few lines of log of each server during lag.

Server A log (Clients IPs modified):

Apr 26 14:34:03 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:03 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:03 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:03 2.247.194.123:0 rejected proxy/trojan: not a valid user Apr 26 14:34:03 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:03 2.247.194.123:0 rejected proxy/trojan: not a valid user Apr 26 14:34:09 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:09 5.108.69.130:0 rejected proxy/trojan: not a valid user Apr 26 14:34:11 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:11 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:11 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:11 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:12 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:12 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:12 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:12 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:13 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:13 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:13 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:13 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:13 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:13 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:13 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:13 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:14 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:14 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:15 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:15 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:15 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:15 2.247.194.123:0 rejected proxy/trojan: not a valid user Apr 26 14:34:15 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:15 2.247.194.123:0 rejected proxy/trojan: not a valid user Apr 26 14:34:15 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:15 2.247.194.123:0 rejected proxy/trojan: not a valid user Apr 26 14:34:15 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:15 5.108.69.130:0 rejected proxy/trojan: not a valid user Apr 26 14:34:15 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:15 5.108.69.130:0 rejected proxy/trojan: not a valid user Apr 26 14:34:15 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:15 5.108.69.130:0 rejected proxy/trojan: not a valid user Apr 26 14:34:15 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:15 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:16 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:16 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:17 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:17 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:18 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:18 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:19 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:19 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:19 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:19 5.117.91.193:0 rejected proxy/trojan: not a valid user Apr 26 14:34:21 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:21 89.199.171.114:0 rejected proxy/trojan: not a valid user Apr 26 14:34:22 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:22 89.199.171.114:0 rejected proxy/trojan: not a valid user Apr 26 14:34:23 some.hostname.here XMPlus[69668]: 2023/04/26 14:34:23 89.199.171.114:0 rejected proxy/trojan: not a valid user

Server B log (Clients IPs modified):

Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 5.210.42.143:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:38 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:38 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id proxy/vless/encoding: invalid request user id Apr 26 14:56:39 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:39 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:39 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:39 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:39 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:39 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:39 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:39 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:39 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:39 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:39 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:39 89.199.64.30:0 rejected proxy/vless/encoding: invalid request user id Apr 26 14:56:39 some.hostname.here XMPlus[153476]: 2023/04/26 14:56:39 5.210.42.143:0 rejected proxy/vless/encoding: invalid request user id

Server C log (Clients IPs modified):

Apr 26 15:27:47 some.hostname.here XMPlus[590]: 2023/04/26 15:27:47 5.121.30.93:39630 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:27:51 some.hostname.here XMPlus[590]: 2023/04/26 15:27:51 95.162.187.207:63841 rejected proxy/vmess/encoding: invalid user > replayed request Apr 26 15:27:51 some.hostname.here XMPlus[590]: 2023/04/26 15:27:51 5.121.30.93:39630 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:27:54 some.hostname.here XMPlus[590]: 2023/04/26 15:27:54 5.121.30.93:39630 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:27:56 some.hostname.here XMPlus[590]: 2023/04/26 15:27:56 5.127.158.87:54179 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:27:56 some.hostname.here XMPlus[590]: 2023/04/26 15:27:56 5.127.158.87:54179 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:27:58 some.hostname.here XMPlus[590]: 2023/04/26 15:27:58 5.121.30.93:39630 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:15 some.hostname.here XMPlus[590]: 2023/04/26 15:28:15 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:16 some.hostname.here XMPlus[590]: 2023/04/26 15:28:16 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:16 some.hostname.here XMPlus[590]: 2023/04/26 15:28:16 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:17 some.hostname.here XMPlus[590]: 2023/04/26 15:28:17 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:21 some.hostname.here XMPlus[590]: 2023/04/26 15:28:21 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:22 some.hostname.here XMPlus[590]: 2023/04/26 15:28:22 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:26 some.hostname.here XMPlus[590]: 2023/04/26 15:28:26 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:27 some.hostname.here XMPlus[590]: 2023/04/26 15:28:27 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:27 some.hostname.here XMPlus[590]: 2023/04/26 15:28:27 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:28 some.hostname.here XMPlus[590]: 2023/04/26 15:28:28 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:32 some.hostname.here XMPlus[590]: 2023/04/26 15:28:32 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:33 some.hostname.here XMPlus[590]: 2023/04/26 15:28:33 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:37 some.hostname.here XMPlus[590]: 2023/04/26 15:28:37 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:38 some.hostname.here XMPlus[590]: 2023/04/26 15:28:38 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:43 some.hostname.here XMPlus[590]: 2023/04/26 15:28:43 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:48 some.hostname.here XMPlus[590]: 2023/04/26 15:28:48 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:28:50 some.hostname.here XMPlus[590]: 2023/04/26 15:28:50 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:29:17 some.hostname.here XMPlus[590]: 2023/04/26 15:29:17 2.147.44.50:29835 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:29:28 some.hostname.here XMPlus[590]: 2023/04/26 15:29:28 5.127.186.242:47738 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:29:28 some.hostname.here XMPlus[590]: 2023/04/26 15:29:28 5.127.186.242:47738 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found Apr 26 15:29:28 some.hostname.here XMPlus[590]: 2023/04/26 15:29:28 5.127.186.242:47738 rejected proxy/vmess/encoding: invalid user > proxy/vmess: Not Found

CPU consumption during incident

image

Any idea? or did I misconfiguration?

letmefind commented 1 year ago

Update : LAG time is correlated to /etc/XMPlus/config.yml config.yml "UpdatePeriodic: 60" where report Online IPs/Users Deleted& Added so What API update periods do here if has been set on 120s? sure thing it's checking server functionality, but obviously any changes in Api time will affect the number of Online IPs as well, so has to define the same value in both configs? meant in YML file & API settings.

peter2186668355 commented 1 year ago

me too！！！

subzero9319 commented 1 year ago

Me to image

letmefind commented 1 year ago

Regarding issue of updating api , latest update shows some changes in configuration. Referring to XMPlus Backend Installation - Backend "UpdatePeriodic : 60" has been removed from YML config file and also I noticed Update Period at API Settings has been removed in v20230504 . Regardless of changes in lates version the XMPlus service in backend is updating every a min causes CPU spike which is expected but filling 400% CPU consumption during update is an issue .

Fortunately, despite of heading plan to use WS, invalid request user id has been sorted in by changing GRPC to WS. seems some CDN manipulating GRPC SSL packet so the requests toward servers would be drop in web servers and ended up to the invalid request.

xcode75 commented 1 year ago

noted, cpu issue already been solved, will updated tomorrow