search

ErdbeerbaerLP / DiscordIntegration-Forge

Forge version of the Discord Integration mod
https://modrinth.com/plugin/dcintegration
MIT License
99 stars 54 forks source link

[BUG] Massive ram spike to the point of server crashing #326

Open jpwanabe opened 2 years ago

jpwanabe commented 2 years ago

What happened?

What should happen?

Steps to reproduce

Version

Platform

Mod version: dcintegration-forge-2.3.2-1.16.jar

Config file -https://pastebin.com/k2XDcKpi REMOVE YOUR BOT TOKEN BEFORE UPLOADING!!!

Crash report if available (use https://hastebin.com) -Not a crash but some of the error https://pastebin.com/TBu0qFK5

Additional info

This happens randomly. I've seen it happen on multiple versions but this was the first time I watched it happen in console. This plugin is the best I've seen for discord and Minecraft so I want to keep it.

ErdbeerbaerLP commented 2 years ago

I will look into this. Usually an ratelimit (which is what seems to happen here) should not use that much RAM. For some reason it overuses this webhook. Can you try if this bug also happens with webhook mode disabled?

jpwanabe commented 2 years ago

If I could force this to happen I definitely would. But I will turn it off for a while and see if it happens

On Tue, Aug 24, 2021, 6:27 PM ErdbeerbaerLP @.***> wrote:

I will look into this. Usually an ratelimit (which is what seems to happen here) should not use that much RAM. For some reason it overuses this webhook. Can you try if this bug also happens with webhook mode disabled?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ErdbeerbaerLP/DiscordIntegration-Forge/issues/326#issuecomment-905018612, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGO4U74Y56HPATJ3F2YLRDT6QMGTANCNFSM5CVNCERA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

jpwanabe commented 2 years ago

So I've been running the plug-in without webhooks and it has still happens multiple times. It seems to be when a player is trying to whitelist themselves and is messing up the command. I think if you spam the bot with wrong commands over and over again it causes this. I've only been able to do it once myself since trying. And it only seems to work if the server already has five or more players on it.

ErdbeerbaerLP commented 2 years ago

Interesting. Will need to look at it on some public server... Currently while changing whitelisting stuff, I got an bug which causes whitelisting to fail for no reason (no errors in log). Will investigate this even more as soon as I get the chance to do that

jpwanabe commented 2 years ago

I may have an idea with the whitelisting. I have had players fail if their minecraft name had any discord formatting commands in it. With the old version or the new one. I understood why the /whitelist "mc name" would fail, but not so sure why !link # would. Maybe the bot response telling you your name?

Kytech commented 2 years ago

@ErdbeerbaerLP I've been experiencing a similar issue on my server using the Forge version on mc 1.16 where RAM usage climbs to the point that it crashes the server due to the GC having to work overtime (by the point Xmx is reached). Raising Xmx just increases the amount of time needed until this happens regardless of if the server is idle or has players, so it definitely seems like a memory leak. I am using the most current version of the mod for 1.16 as of this comment.

I haven't necessarily been able to directly link the issue to messages not sending to discord, though I took a heap dump of the server when it crashed and it is showing the rate limiter to be consuming the bulk of the server's RAM usage (~3.5 out of 4GB total allocated) to the point where it over-stresses the Garbage Collector, causing the crash. The pattern I have noticed is that it starts building up gradually over time, even when the server is idle, though I'll have to see if messages aren't getting through when the RAM usage starts to run high again. I'll attach a report I generated from the heap dump to give an idea of how many requests are getting backed up in memory in addition to a copy of my GC logs since it gives a good illustration of how the memory gradually increases over time.

If you're needing further information, I could send you snapshots and other reports of the heap dump if you specify which classes you care about seeing in the dump to keep the amount of processing minimal, or if that doesn't suffice, I can send a copy of the whole heap dump (Indexing the full dump took about 8GB of RAM allocated to eclipse MAT if we want to go that direction, though I still have my index files around if that helps.)

Let me know if there's anything else I can do to help. You've certainly got the most feature-rich discord mod for forge that is still maintained and it has all the features I'm needing for my server, though I may have to remove the mod until we can get this fixed up.

heapdump.20210929.122708.1227838.0001_Leak_Suspects.zip verbosegc.20210928.225009.1227838.txt

Kytech commented 2 years ago

For context, I have been running the server on OpenJ9 when I generated these logs and the heap dump, though the issue occurs in a hotspot-based build of Java as well. I've tried just about every set of garbage collection tweaks as I can think of with multiple different algorithms on both hotspot and openj9 and the same behavior ultimately plays out for me.

Kytech commented 2 years ago

Note sure if this is connected to #283 in any way or if that issue is contributing to the build-up of requests in the rate limiter's memory, but I do experience extreme lag in the server as it approaches the crash point. In my case, the lag seems to be caused by the very frequent garbage collection as the memory fills up.

Kytech commented 2 years ago

An update: May have found a cause for the build-up of rate-limited requests. Apparently, this mod has been logging command block output to discord, even if the commands do not send anything to the in-game server chat. We have a command block circuit that's constantly running at spawn of my server, which might be the cause of the message build-up and extreme RAM usage. Is there some config option that sends all command block output to the server or is this a bug where it's picking up on everything a command block outputs regardless of if it's sent to the in-game chat or not? Found the config option that logs commands to discord. Will try disabling it to see if that solves my issue. If it does, I'll probably open up a feature request to have an option where the command logging config can have some way to exclude command blocks and only log commands sent by players.

Kytech commented 2 years ago

@ErdbeerbaerLP I have an update on this. Disabling command logging seems to prevent the memory leak from occurring. However, I do not know if the bug only occurs if the channel ID for command logging is for a channel that does not exist (had a nonexistent channel in my case) or if it also occurs with cases where the channel exists. May want to look into whether the issue can be re-created when the channel exists. Command block and data pack commands were getting logged by this mod while I was troubleshooting my server.

@jpwanabe you might want to try disabling command logging to see if that prevents the RAM spike for you.

jpwanabe commented 2 years ago

More info on this crash. If a player types the !link # command in the chat the bot is using the bot goes offline on discord almost 100% of the time. Then the ram starts to spike. And crash soon after.