1.16.5: Errors with BungeeControl.

FyreMCOfficial commented 2 years ago

Hello!

Unfortunately, we faced a strange problem when many players would start playing on the Bungee network. 400+ I will try to describe in as much detail as possible what our own tests have repeatedly confirmed. I will describe the phenomenon itself and the steps we have now taken to eliminate it temporarily. I attach several error messages.

The error phenomenon: Players are almost always dropped with a "Proxy lost connection" message within 1 minute. For some reason, it only starts with 350-400 players, but from that number, the problem keeps popping up.

What you need to know about ChatControl settings: ChatControl operates in bungee mode with BungeeControl. Unlike ChatControl PRO, mysql is required.

Connection lost error message from SPIGOT. "Player1234 lost connection: Internal Exception: io.netty.handler.codec.DecoderException: java.io.IOException: Packet 0/0 (PacketPlayInTeleportAccept) was larger than I expected, found 41787 bytes extra whilst reading packet 0"

Error message on BUNGEE page. We get some such messages, not very often.

"[Player123] disconnected with: §fInternal Exception: io.netty.handler.codec.DecoderException: java.io.IOException: Payload may not be larger than 32767 bytes"

And we get that too, also not very often.

"13:36:29 [INFO] BungeeControl-Red 3.8.5 encountered a ReaderException! Please check your error.log and report this issue with the information in that file.
13:36:29 [INFO] unacceptable code point ' ' (0x0) special characters are not allowed
in "'reader'", position 0
13:36:29 [INFO]     at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:211)
13:36:29 [INFO]     at org.yaml.snakeyaml.reader.StreamReader.ensureEnoughData(StreamReader.java:176)
13:36:29 [INFO]     at org.yaml.snakeyaml.reader.StreamReader.ensureEnoughData(StreamReader.java:171)
13:36:29 [INFO]     at org.yaml.snakeyaml.reader.StreamReader.peek(StreamReader.java:126)
13:36:29 [INFO]     at org.yaml.snakeyaml.scanner.ScannerImpl.scanToNextToken(ScannerImpl.java:1177)
13:36:29 [INFO]     at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:287)
13:36:29 [INFO]     at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:227)
13:36:29 [INFO]     at org.yaml.snakeyaml.parser.ParserImpl$ParseImplicitDocumentStart.produce(ParserImpl.java:195)
13:36:29 [INFO]     at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
13:36:29 [INFO]     at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:148)
13:36:29 [INFO]     at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:107)
13:36:29 [INFO]     at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:139)
13:36:29 [INFO]     at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:524)
13:36:29 [INFO]     at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:518)
13:36:29 [INFO]     at net.md_5.bungee.config.YamlConfiguration.load(YamlConfiguration.java:111)
13:36:29 [INFO]     at net.md_5.bungee.config.YamlConfiguration.load(YamlConfiguration.java:79)
13:36:29 [INFO]     at org.mineacademy.bungeecontrol.lib.bfo.settings.ConfigInstance.reload(YamlConfig.java:1597)
13:36:29 [INFO]     at org.mineacademy.bungeecontrol.lib.bfo.settings.ConfigInstance.save(YamlConfig.java:1574)
13:36:29 [INFO]     at org.mineacademy.bungeecontrol.lib.bfo.settings.YamlConfig.save(YamlConfig.java:428)
13:36:29 [INFO]     at org.mineacademy.bungeecontrol.ServerCache.registerPlayer(ServerCache.java:87)
13:36:29 [INFO]     at org.mineacademy.bungeecontrol.listener.SwitchListener.onDisconnect(SwitchListener.java:116)
13:36:29 [INFO]     at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
13:36:29 [INFO]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
13:36:29 [INFO]     at java.lang.reflect.Method.invoke(Method.java:498)
13:36:29 [INFO]     at net.md_5.bungee.event.EventHandlerMethod.invoke(EventHandlerMethod.java:29)
13:36:29 [INFO]     at net.md_5.bungee.event.EventBus.post(EventBus.java:46)
13:36:29 [INFO]     at bungeepluginmanager.ModifiedPluginEventBus.post(ModifiedPluginEventBus.java:33)
13:36:29 [INFO]     at net.md_5.bungee.api.plugin.PluginManager.callEvent(PluginManager.java:402)
13:36:29 [INFO]     at net.md_5.bungee.connection.UpstreamBridge.disconnected(UpstreamBridge.java:116)
13:36:29 [INFO]     at net.md_5.bungee.netty.HandlerBoss.channelInactive(HandlerBoss.java:57)
13:36:29 [INFO]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
13:36:29 [INFO]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
13:36:29 [INFO]     at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
13:36:29 [INFO]     at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
13:36:29 [INFO]     at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
13:36:29 [INFO]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
13:36:29 [INFO]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
13:36:29 [INFO]     at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
13:36:29 [INFO]     at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
13:36:29 [INFO]     at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)
13:36:29 [INFO]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
13:36:29 [INFO]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
13:36:29 [INFO]     at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
13:36:29 [INFO]     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
13:36:29 [INFO]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
13:36:29 [INFO]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
13:36:29 [INFO]     at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
13:36:29 [INFO]     at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)
13:36:29 [INFO]     at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
13:36:29 [INFO]     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
13:36:29 [INFO]     at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
13:36:29 [INFO]     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
13:36:29 [INFO]     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
13:36:29 [INFO]     at java.lang.Thread.run(Thread.java:748)
13:36:29 [INFO] §7!-----------------------------------------------------!
13:36:29 [INFO] §cFailed to save data.db
13:36:29 [INFO] §7!-----------------------------------------------------!

The contents of error.log are for BungeeControl.

------------------------------------[ 17.12.2021 13:37:03 ]-----------------------------------
BungeeControl-Red 3.8.5 encountered a ReaderException
Running Aegis Aegis fork and Java 1.8.0_281
Plugins: LiteAnnouncer 6.3.1, reconnect_yaml git:reconnect_yaml:1.15-SNAPSHOT:53695f0:343, LuckPerms 5.2.30, Emoji 1.0, SQLiteDriver 1.0.0, UltimateFriends 2.6.6, PlayerMute 1.0.0, PremiumVanish 2.5.2, BungeeControl-Red 3.8.5, BungeeOnlineTime 6.1, GList 1.0.0, UltimateAutoRestartPassthrough Build 1, WelcomeTitle 1.0.5, BungeeCordMemoryUsage 2.0, cmd_server git:cmd_server:1.17-R0.1-SNAPSHOT:1b489bc:1590, CurseMessenger 0.1, LiteBans 2.6.2, cmd_send git:cmd_send:1.15-SNAPSHOT:53695f0:343, BungeeServerManager 1.1.2-SNAPSHOT, LobbyBalancer 1.0, PlayerVanish 1.0.0, Maintenance 3.0.5, JoinFullServer 1.0, BungeePluginManager 2.0, TP 1.0, cmd_find git:cmd_find:1.15-SNAPSHOT:53695f0:343, MediaAnnounce 1.0.0, Party 1.0, cmd_alert git:cmd_alert:1.15-SNAPSHOT:53695f0:343, PlayerStat 1.0.0
----------------------------------------------------------------------------------------------

More Information: 
Failed to save data.db
ReaderException special characters are not allowed
     at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:211)
     at org.yaml.snakeyaml.reader.StreamReader.ensureEnoughData(StreamReader.java:176)
     at org.yaml.snakeyaml.reader.StreamReader.ensureEnoughData(StreamReader.java:171)
     at org.yaml.snakeyaml.reader.StreamReader.peek(StreamReader.java:126)
     at org.yaml.snakeyaml.scanner.ScannerImpl.scanToNextToken(ScannerImpl.java:1177)
     at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:287)
     at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:227)
     at org.yaml.snakeyaml.parser.ParserImpl$ParseImplicitDocumentStart.produce(ParserImpl.java:195)
     at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
     at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:148)
     at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:107)
     at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:139)
     at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:524)
     at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:518)
     at net.md_5.bungee.config.YamlConfiguration.load(YamlConfiguration.java:111)
     at net.md_5.bungee.config.YamlConfiguration.load(YamlConfiguration.java:79)
     at org.mineacademy.bungeecontrol.lib.bfo.settings.ConfigInstance.reload(YamlConfig.java:1597)
     at org.mineacademy.bungeecontrol.lib.bfo.settings.ConfigInstance.save(YamlConfig.java:1574)
     at org.mineacademy.bungeecontrol.lib.bfo.settings.YamlConfig.save(YamlConfig.java:428)
     at org.mineacademy.bungeecontrol.ServerCache.registerPlayer(ServerCache.java:87)
     at org.mineacademy.bungeecontrol.listener.SwitchListener.onDisconnect(SwitchListener.java:116)
     at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:498)
     at net.md_5.bungee.event.EventHandlerMethod.invoke(EventHandlerMethod.java:29)
     at net.md_5.bungee.event.EventBus.post(EventBus.java:46)
     at bungeepluginmanager.ModifiedPluginEventBus.post(ModifiedPluginEventBus.java:33)
     at net.md_5.bungee.api.plugin.PluginManager.callEvent(PluginManager.java:402)
     at net.md_5.bungee.connection.UpstreamBridge.disconnected(UpstreamBridge.java:116)
     at net.md_5.bungee.netty.HandlerBoss.channelInactive(HandlerBoss.java:57)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
     at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
     at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
     at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
     at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
     at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
     at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)
     at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
     at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
     at java.lang.Thread.run(Thread.java:748)
----------------------------------------------------------------------------------------------

I don't think bungee-side error messages lead to the phenomenon. When the player is dropped by the server, that’s the message I showed from SPIGOT.

Current solution: If I remove BungeeControl, the error message will disappear from both Bungee and Spigot. And we don't have a problem with 1000+ players on the server either.

I know you've already released a new version, but does that solve the problem? It is risky to test on such a network where 1,200 people play daily.

Minecraft Version: 1.16.5 MySQL: true BungeeCord: true

kangarko commented 2 years ago

Wait with the update, I don't think the 3.8.6 will change this. Going to investigate myself in 24 hours. Thanks for using ChatControl.

My feeling is that the packet size we send to synchronize data is too large so the packet overflows. We already mitigate this when a 250-player server reported this problem. I will look into an alternative way to send data asap.

kangarko commented 2 years ago

I've disabled the onReload call that was invoked upon each save. That should mitigate the character error. Another thing you can do is try using the utf8 flag in your startup script as shown in this old article (still actual): https://bukkit.org/threads/startup-script-flag-for-enabling-utf-8.171342/

Regarding the PacketPlayInTeleportAccept packet kick, this is not caused by ChatControl afaik, please report that to Paper/Spigot.

There's going to be a new release tonight but if you want to save work I recommend asking about the teleportaccept packet on Paper/Spigot first before testing that build since it has no changes in that regards.

kangarko commented 2 years ago

Did you ask about this in Paper — is there anything I can assist you with?

We also had a minor change to data cleanup in BungeeControl Red, I encourage you to upgrade now.

kangarko commented 2 years ago

Closing due to inactivity, let us know below if you need anything.

Nexxuser commented 2 years ago

Thank you and sorry for the late reply. (Holidays) I didn't write to Paper because I'll be updating to 1.17 soon to see if it resolves there. I tried the new version today, now with the following error message, it drops players around 400 players as well. unknown .

FyreMCOfficial commented 2 years ago

Ohh, sorry. This is also my profile.

kangarko commented 2 years ago

Trying to simulate 400 online players on my test server to test this.

kangarko commented 2 years ago

@FyreMCOfficial sorry, I was able to get to over 400 players using a simulator, but never experienced this: https://i.imgur.com/9ezB2dc.png

We had a protocol update in the last version, let me know when that happens again --- try to find out what preceeded the kick, such as what commands were typed? Are these players OPs or admins? Does it happen when you hit [X] to remove a chat message ? Etc. anything I can use to dig deeper, thank you.

kangarko / ChatControl-Red

1.16.5: Errors with BungeeControl. #1566