A bug that could cause all players to be kicked from the server!

[CatzMR]

After updating the game in the last two months, my friend's server has the problem of all players being kicked This problem occurs when the server autosaves. The server will suddenly have a TPS of 0 at a certain moment, and then the server will not respond. After about 30 seconds, all players will be automatically kicked out of the server. This problem has been repeated on his server After the server starts getting responses, the server kicks all players off the server. ` [2023/1/2 15:00:25]：32.55273 seconds between inputs from "李耀屑真蓝人" steamid: 76561199433185862 [2023/1/2 15:00:25]：32.55273 seconds between inputs from "二之前Ninomae" steamid: 76561198371784116 [2023/1/2 15:00:25]：32.55273 seconds between inputs from "烧鸡翅" steamid: 76561199198528582 [2023/1/2 15:00:25]：32.55273 seconds between inputs from "江非" steamid: 76561199397418412 [2023/1/2 15:00:25]：32.55469 seconds between inputs from "GCGJgc" steamid: 76561199433880678 [2023/1/2 15:00:25]：32.55469 seconds between inputs from "unturned_2023" steamid: 76561199466881093 [2023/1/2 15:00:25]：32.55469 seconds between inputs from "风起微澜小号" steamid: 76561199270161748 [2023/1/2 15:00:25]：32.60352 seconds between inputs from "ikunsnow" steamid: 76561199434301375 [2023/1/2 15:00:25]：32.55469 seconds between inputs from "我的阿妈每天早上起床给我哨兵充电" steamid: 76561198207995443 [2023/1/2 15:00:25]：32.55469 seconds between inputs from "DjMist" steamid: 76561198872601608 [2023/1/2 15:00:25]：32.55664 seconds between inputs from "emmmm" steamid: 76561199015768856 [2023/1/2 15:00:25]：32.51758 seconds between inputs from "Zheng_Jian" steamid: 76561199196097522 [2023/1/2 15:00:25]：32.51758 seconds between inputs from "我的无人机会放飞密客" steamid: 76561199088738383 [2023/1/2 15:00:25]：32.51758 seconds between inputs from "是柠柒吖" steamid: 76561198956883273 [2023/1/2 15:00:25]：32.51758 seconds between inputs from "七里浜の海" steamid: 76561199026735284 [2023/1/2 15:00:25]：32.51758 seconds between inputs from "Jane Lakitzd" steamid: 76561199082029285 [2023/1/2 15:00:25]：32.51953 seconds between inputs from "赛博Miku会梦见电子天依吗" steamid: 76561198252114996 [2023/1/2 15:00:25]：32.51953 seconds between inputs from "大风歌12" steamid: 76561198370687682 [2023/1/2 15:00:25]：32.51953 seconds between inputs from "c6h6o8" steamid: 76561199178829363 [2023/1/2 15:00:25]：32.51953 seconds between inputs from "hanpi" steamid: 76561199080466068 [2023/1/2 15:00:25]：32.52148 seconds between inputs from "cheems114514idk" steamid: 76561199371605826 [2023/1/2 15:00:25]：32.52148 seconds between inputs from "SAINT-14" steamid: 76561198339632825 [2023/1/2 15:00:25]：32.52148 seconds between inputs from "qqqqqqq" steamid: 76561199446257318 [2023/1/2 15:00:25]：32.52148 seconds between inputs from "30岁性感蕾丝少妇" steamid: 76561198299743725 [2023/1/2 15:00:25]：32.52148 seconds between inputs from "BOT aimoer" steamid: 76561199194385983 [2023/1/2 15:00:25]：32.52148 seconds between inputs from "你先别急先让我急" steamid: 76561199418986403

[2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199371605826 Name: cheems114514idk Character: cheems114514idk [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199080466068 Name: hanpi Character: hanpi [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199433185862 Name: 李耀屑真蓝人 Character: 李耀屑小号 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199397418412 Name: 江非 Character: 江非 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199270161748 Name: 风起微澜小号 Character: 风起微澜小号 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199178829363 Name: c6h6o8 Character: c6h6o8 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561198299743725 Name: 30岁性感蕾丝少妇 Character: 城暮 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199433880678 Name: GCGJgc Character: GCGY臣 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561198339632825 Name: SAINT-14 Character: daxina [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199466881093 Name: unturned_2023 Character: 发狙者小小小小号 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561198252114996 Name: 赛博Miku会梦见电子天依吗 Character: 世界LOST失落 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561198956883273 Name: 是柠柒吖 Character: 是柠柒吖 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561198207995443 Name: 我的阿妈每天早上起床给我哨兵充电 Character: Zany_MKB [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199082029285 Name: Jane Lakitzd Character: Jane Lakitzd [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561198370687682 Name: 大风歌12 Character: 大风歌 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199196097522 Name: Zheng_Jian Character: Zheng_Jian [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199015768856 Name: emmmm Character: emmm098 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199026735284 Name: 七里浜の海 Character: 白给日记 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561198371784116 Name: 二之前Ninomae Character: 二之前Ninomae [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199434301375 Name: ikunsnow Character: ikunsnow [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199418986403 Name: 你先别急先让我急 Character: 翰泽 [2023/1/2 15:00:25]：Disconnecting: PlayerID: 76561199198528582 Name: 烧鸡翅 Character: 小陌 [2023/1/2 15:00:25]：Player finished session: 76561199080466068 [2023/1/2 15:00:25]：Player finished session: 76561199433185862 [2023/1/2 15:00:25]：Player finished session: 76561198207995443 [2023/1/2 15:00:25]：Player finished session: 76561199371605826 [2023/1/2 15:00:25]：Player finished session: 76561199088738383 [2023/1/2 15:00:25]：Player finished session: 76561199270161748 [2023/1/2 15:00:25]：Player finished session: 76561199178829363 [2023/1/2 15:00:25]：Player finished session: 76561198299743725 [2023/1/2 15:00:25]：Player finished session: 76561199466881093 [2023/1/2 15:00:25]：Player finished session: 76561199433880678 [2023/1/2 15:00:25]：Player finished session: 76561199397418412 [2023/1/2 15:00:25]：Player finished session: 76561198252114996 [2023/1/2 15:00:25]：Player finished session: 76561198339632825 [2023/1/2 15:00:25]：Player finished session: 76561198370687682 [2023/1/2 15:00:25]：Player finished session: 76561198872601608 [2023/1/2 15:00:25]：Player finished session: 76561199082029285 [2023/1/2 15:00:25]：Player finished session: 76561199015768856 [2023/1/2 15:00:25]：Player finished session: 76561198956883273 [2023/1/2 15:00:25]：Player finished session: 76561199196097522 [2023/1/2 15:00:25]：Player finished session: 76561199026735284 [2023/1/2 15:00:25]：Player finished session: 76561199446257318 [2023/1/2 15:00:25]：Player finished session: 76561198371784116 [2023/1/2 15:00:25]：Player finished session: 76561199434301375 [2023/1/2 15:00:25]：Player finished session: 76561199198528582 [2023/1/2 15:00:25]：Player finished session: 76561199418986403 [2023/1/2 15:00:25]：Player finished session: 76561199194385983` After the server starts getting responses, the server kicks all players off the server. We recently added in-game event detection to take this idea a step further, as players are building massively when the server is unresponsive! I think it may be caused by memory overflow at a certain time, and the cache cannot be saved in time. This problem occurs when the server autosaves.

[SDGNelson]

If the main thread is blocked for 30s it makes sense that everyone loses connection. Vanilla also has logging for this "long delay between Updates". Unfortunately if autosave is the culprit there is not much you can do unless there are plugins patching save/load that might be slowing them down - e.g. Avi does this. Or if disk speed is the problem it might help to use an SSD. One of the future rewrites I intend to do is copying savedata to a different thread that will handle serializing and saving it rather than doing those on the main thread.

[CatzMR]

We are using NVMe hard drives, and there is no problem with the speed of the hard drives.

[CatzMR]

save/load that might be slowing them down - e.g. Avi does this.

I don't quite understand what this means....

[GazziFX]

I don't quite understand what this means....

AviRockets plugins is bad

[CatzMR]

Not using AviRocket， I am using the official latest RocketMod.

[CatzMR]

The auto-save function of the latest official RocketMod will trigger the above problem.

[GazziFX]

Can you attach list of all your plugins?

[CatzMR]

At the beginning, there were only Uconomy, ZaupShop, and ZaupUconomyEssentials, and the problem existed, and other plug-ins were added later.

你能附上你所有插件的列表吗？

[fahrettinenes]

At the beginning, there were only Uconomy, ZaupShop, and ZaupUconomyEssentials, and the problem existed, and other plug-ins were added later.

你能附上你所有插件的列表吗？

Hello! if i recall correctly (and i think i do) Uconomy plugin is running sql queries on the main thread. ZaupUconomyEssentials plugin is also doing this. e.g. when someone kill 5 zombie in the same time -and if you open the zaupuconomyessentials' zombie reward of course- your server's main thread will be blocked until uconomy process these queries. I also recommend using a local mysql server for uconomy (no clue about if you already do that)

I also got hired like couple weeks ago to fix this ZaupUconomyEssentials' lag problem. I could recommend calling database actions from off-thread (e.g. Task.Run or use UniTasks). -or just edit the uconomy ¯\(ツ)/¯ (it will cause bugs on the plugins that use unity functions on the IncreaseBalance method's prefix -harmony one- :D)-

[fahrettinenes]

Also check the issue without ZaupUconomyEssentials (it can cause this issue while players are playing on your server), BaseClustering (same as uconomyessentials), and NGE_GlobalBan (never heard of this plugin; maybe it makes a sql query on each player connection, who knows? :D)

I'm also not sure about the issue. They are just some guesses; I can't really say more things without more information. Like I said, these are just some guesses. But it will be interesting if some of these plugins block the main thread for more than 30 seconds.

[CatzMR]

Also check the issue without ZaupUconomyEssentials (it can cause this issue while players are playing on your server), BaseClustering (same as uconomyessentials), and NGE_GlobalBan (never heard of this plugin; maybe it makes a sql query on each player connection, who knows? :D)

I'm also not sure about the issue. They are just some guesses; I can't really say more things without more information. Like I said, these are just some guesses. But it will be interesting if some of these plugins block the main thread for more than 30 seconds.

It should have nothing to do with the plug-ins. This problem has already appeared without these plug-ins. After adding these plug-ins, the problem has not aggravated.

[fahrettinenes]

It should have nothing to do with the plug-ins. This problem has already appeared without these plug-ins. After adding these plug-ins, the problem has not aggravated.

It should be about dedicated's hardware then. I have no clue, good luck on your problem :c