search

FTBTeam / FTB-Interactions

14 stars 10 forks source link

Memory Leak / Extreme Memory Usage in Main Menu after Entering and Exiting a World #477

Closed liketechnik closed 5 years ago

liketechnik commented 5 years ago

Edited version after I found out where the leak occurs:

Modpack Version:

1.3.0

Bug Report:

Minecraft starts to fill up my 16GB in less than 15 minutes, although maximum usage is set to 7GB via Java options. Some details to this:

Expected Behavior/Suggestions:

Used memory does not increase in main menu after exiting a world.

Is It Repeatable? Steps to Reproduce:

  1. Set maximum memory to prefered amount (7GB in this case).
  2. Start playing.
  3. Wait and kill MC fast enough before your PC locks up.
  4. Exit the world.
  5. Exit MC before your PC locks up.

Link to Log or Crash File Paste**:

None, as MC does not crash. There's nothing about the main menu in the log, just the usual messages for unloading a world.

Mod/s Affected:

Some main menu tweaker mod?!

Known Fix:

Is it OS (linux) or Java version related (OpenJDK 1.8.0_202)?

nowifi4u commented 5 years ago

Issue is probably in Garbage Collector. Custom launch parameters should fix the issue. Parameters tested on multiple big modpacks: -XX:+UseG1GC -Dsun.rmi.dgc.server.gcInterval=2147483646 -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 -XX:G1ReservePercent=20 -XX:MaxGCPauseMillis=25 -XX:G1HeapRegionSize=32M -XX:+OptimizeStringConcat -XX:+UseLargePages -XX:+UseFastAccessorMethods

liketechnik commented 5 years ago

Thank you for mentioning these, totally forgot to mention that I have set up custom launch parameters. I'm gonna try out the one's you posted in a moment.

Anyways, here's what I used in the past: -d64 -XX:+DisableExplicitGC -XX:MaxTenuringThreshold=15 -XX:-UseGCOverheadLimit -XX:+UseBiasedLocking -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:+UseCompressedOops -XX:+OptimizeStringConcat -XX:ReservedCodeCacheSize=1024m -XX:+UseCodeCacheFlushing -XX:SoftRefLRUPolicyMSPerMB=20000 -XX:+AggressiveOpts -XX:MaxGCPauseMillis=30 -XX:+UseConcMarkSweepGC -XX:-UseAdaptiveSizePolicy -XX:ParallelGCThreads=6

liketechnik commented 5 years ago

Ok, so I tried it with your options and loading the world was normal (well the ram usage increased to 13GB, but I guess that's 'normal' for big modpacks), but after quitting the world and going back to the main menu I could see the ram usage increasing at a rate of 0.1GB/sec. So some thing (probably some sort of main menu tweaker mod) is leaking a lot of memory after entering and exiting a world.

liketechnik commented 5 years ago

Just checked, this issue exists in the 1.0.0 (first published version) version too, so probably in all versions.

Saereth commented 5 years ago

I cant reproduce this at all, also 13gb is way too much ram to allocate, no more than 6 or 7 even with shaders/texture pack

liketechnik commented 5 years ago

I can't reproduce it anymore too, it's vanished. I don't really know what to say about this, because in my opinion, it's still a big risk, especially because I'm unable to name a cause for this, so it could possibly happen anytime again and crash the computer.

Regarding the amount of ram to allocate: I ran the pack with 6GB max. allocation allowed when the leak happened. The thing is, you can only control the heap size, and as MC creates a lot of threads (around 60 when in a world), they each take some memory, so I think it's normal that the total value used by MC is higher than those 6G (and the memory used is still higher than the 6G, although the leak didn't happen anymore).

I'll close this issue for now, as there is no way at the moment to reproduce the issue and hope I won't have to open it again.

liketechnik commented 5 years ago

Sorry to be so noisy about it, but it just happened again. I can just tell you that it happens randomly, the ram usage doubles from around 10GB to 20GB in next to time (from one moment to the next everything is completely frozen) and that it does not happen in neither FTB Sky Adventures or Odyssey if that helps. There is nothing in the logs (I can append it if you want to take a look at the options/system specs and log) and VisualVM (I had it attached just for the case it could crash again) didn't show me anything that made me suspicious (well, it showed me a heap size of 5-6GB and the normal, absurd amount of threads minecraft uses, 60 in the running world, 30 in the main menu), although I'm not sure it worked correctly, because the whole system was frozen during the crash.

I know that this information is probably far from helpful and this crash sounds horribly unrealistic, but I really wish it wouldn't happen or I had a solution for it.

Saereth commented 5 years ago

yeah Idk... we'll leave this open anyway hopefully someone else can contribute some info if others are experiencing it

KAfable commented 5 years ago

Still not too sure sure what might be causing this - the mods that affect main menu are

Torrenal commented 5 years ago

Would heap dumps be safe to share, or will those contain account info secrets? the JDK command jmap can get a heap dump from a running java process, which would allow the person with the problem to forward the necessary details to the person with the skillset to troubleshoot... but I'd only go down that road in this forum if one can be sure secrets (like the minecraft account user/password) don't get shared.

For memory leaks in Java, OS becomes very relevant, and with recent Java builds, so does the java version. Oracle has been tackling memory issues in some of their recent builds, including long standing Mac-only memory leaks. With any updates that fix memory leaks, they can (and have) introduced new bugs that do the opposite.

Edit: Hindsight applying itself here ... The main menu doesn't have any options to show java heap usage, so far as I'm aware . . . so the user is most likely watching task manager or OS specific equievelent, which reports a sum of all memory pools (heap,native,etc) The JVM args supplied don't indicate a heap size, but other comments/instructions indicate a cap on heap size of 7GB. Is Java is limited to 7GB of heap memory but still using 16GB of memory? That sounds like a leak of native resources. I've limited experience with native leaks (I know enough about them to know to not want to touch them). They can be indicated in a normal heap dump, but can be fiendishly easy to miss. Watch for weak/phantom references, some frameworks I've battled with clean up the native resources when the java classes using them are garbage collected. //Yama

Gunner76th commented 5 years ago

@Torrenal If that program works as I believe you are trying to explain it, and dumping the data to a human readable file, then all one would have to do is simply run a search on the created file for their minecraft username/password (Which should not be in the heap as the authentication at the client is performed by comparing a generated token, not a username/password unless ReAuth is used) and then scrub/obfuscate that information from the file before uploading. Unless ReAuth is used, the username and password is never seen by the actual client, as that is handled by the launcher (either Native or Jar. The Twitch client in this context is not counted). The launcher passes the username and password to the Mojang authentication servers, once authenticated it responds back with an authentication token. That token is handed over to the client during pack startup. The only time the username and password are seen at the client level is if you use ReAuth to "refresh/renew" the authentication token.

Torrenal commented 5 years ago

Re heap dumps: They consist of a copy of the main memory pool of a Java application, put onto disk. Most anything a Java program has in memory, is in that heap. Human readable? Some of it. If Minecraft handles passwords well, the password won't be in the file in plaintext but would an encrypted form or account tokens exist? possibly...

Nothing stands out to me other than a probable thread leak (probably minor? Especially if it's a 64-bit JVM).

If possible, check OS and graphics card for affected users. Long shot: affected users might try updating video drivers - I doubt it'll help, but Java video libraries does make use of native memory.

I'll keep an eye open for this myself on my own system (I'm actually staying primed to hunt a different memory leak I saw a week or so back, should it show up for me again). Otherwise, I've looked about as far as I'm prepared to without a reliable recreate or example in front of me.

Research notes follow...

Configured for 8GB heap (Min/Max both at 8GB) Loaded and unloaded a single player creative world several times. Memory footprint (per Windows Task Manager) grew some each time, but resumed normal porportions on garbage collection. As heap size is fixed, memory pools other than heap must be growing/shrinking in response to changes within heap.

The unfixed Oracle-Java memory leak I am aware of (JDK-8222212) does not apply to Minecraft. I am unsurprised by this.

Review of weak/phantom references turns up crypto related (TLS, etc) and graphics related uses from within the Java framework. I'd say that leaves graphics drivers on the table as a possible contributor - except I'm unaware of how Minecraft responds to different graphics hardware. (For JavaFX, I've sometimes had to disable hardware graphics acceleration features for some low end Intel graphics adapters to correct crashes)

Testing was not exhaustive or intended to stress, merely to give myself a feel for how things behaved. Memory use had no red flags - based on how I was using it, 3GB heap would probably be enough (remember - near empty world with just the initial islands - actual gameplay will probably require more)

Each time I returned to the menu from a world, I had 3-4 extra threads running - possible thread leak of Netty Server and Client IO threads? Probably not related to the memory problem but eh, leaks are leaks, plugging them can solve unexpected problems.

Mahadbaloch commented 1 year ago

Several mods can cause this. I play my own modpack and its virtually impossible to tell what is causing this for me. Its likely a common issue.(not related to this modpack)