Thread leak causing server to crash

Jaren8r commented 1 week ago

Describe the bug

After running our server for a while with AdvancedSeasons on it, we start getting resource limit errors and the entire server crashes even before we hit our allocated memory limit. When profiling the server, I see that there are a ton of parked threads belonging to AdvancedSeasons that all have the same stack trace:

   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21.0.3/Native Method)
    - parking to wait for  <0x0...> (a java.util.concurrent.CompletableFuture$Signaller)
    at java.util.concurrent.locks.LockSupport.park(java.base@21.0.3/LockSupport.java:221)
    at java.util.concurrent.CompletableFuture$Signaller.block(java.base@21.0.3/CompletableFuture.java:1864)
    at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.3/ForkJoinPool.java:3780)
    at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.3/ForkJoinPool.java:3725)
    at java.util.concurrent.CompletableFuture.waitingGet(java.base@21.0.3/CompletableFuture.java:1898)
    at java.util.concurrent.CompletableFuture.join(java.base@21.0.3/CompletableFuture.java:2117)
    at net.minecraft.server.level.ChunkProviderServer.a()
    at net.minecraft.world.level.World.a()
    at net.minecraft.world.level.World.a_()
    at org.bukkit.craftbukkit.v1_20_R3.block.CraftBlock.getType()
    at net.advancedplugins.seasons.visuals.type.FallingLeafVisual.createFallingLeaves(FallingLeafVisual.java:68)
    at net.advancedplugins.seasons.visuals.type.FallingLeafVisual.tick(FallingLeafVisual.java:20)
    at net.advancedplugins.seasons.visuals.VisualsHandler$$Lambda/0x00007fbfd75c4678.accept(Unknown Source)
    at java.util.HashMap$KeySet.forEach(java.base@21.0.3/HashMap.java:1016)
    at net.advancedplugins.seasons.visuals.VisualsHandler.lambda$new$0(VisualsHandler.java:28)
    at net.advancedplugins.seasons.visuals.VisualsHandler$$Lambda/0x00007fbfd663d000.run(Unknown Source)
    at org.bukkit.craftbukkit.v1_20_R3.scheduler.CraftTask.run()
    at org.bukkit.craftbukkit.v1_20_R3.scheduler.CraftAsyncTask.run()
    at com.destroystokyo.paper.ServerSchedulerReportingWrapper.run()
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.3/ThreadPoolExecutor.java:1144)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.3/ThreadPoolExecutor.java:642)
    at java.lang.Thread.runWith(java.base@21.0.3/Thread.java:1596)
    at java.lang.Thread.run(java.base@21.0.3/Thread.java:1583)

The number of threads continue to grow until the server crashes:

---- Minecraft Crash Report ----
// I feel sad now :(

Time: 2024-06-18 15:37:04
Description: Exception in server tick loop

java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
    at java.base/java.lang.Thread.start0(Native Method)
    at java.base/java.lang.Thread.start(Thread.java:1553)
    at java.base/java.lang.System$2.start(System.java:2577)
    at java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152)
    at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953)
    at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
    at java.base/java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:754)
    at org.bukkit.craftbukkit.v1_20_R3.scheduler.CraftAsyncScheduler.mainThreadHeartbeat()
    at org.bukkit.craftbukkit.v1_20_R3.scheduler.CraftScheduler.mainThreadHeartbeat()
    at net.minecraft.server.MinecraftServer.b(MinecraftServer.java)
    at net.minecraft.server.dedicated.DedicatedServer.b(DedicatedServer.java)
    at net.minecraft.server.MinecraftServer.a(MinecraftServer.java)
    at net.minecraft.server.MinecraftServer.w(MinecraftServer.java)
    at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java)
    at java.base/java.lang.Thread.run(Thread.java:1583)

How to reproduce

Unfortunately I was not able to reproduce this issue on a testing server by myself, so I don't know how to reproduce this. I have attached our server info+config, as well as a Spark and a jstack showing the massive amount of threads.

Server: Pufferfish+ version git-PufferfishPlus-45 (MC: 1.20.4) (Implementing API version 1.20.4-R0.1-SNAPSHOT) (Git: eb69634 on HEAD) Configuration: archive-2024-06-19T152807+0100.tar.gz

Screenshots / Videos

Spark: https://spark.lucko.me/zvVbNIRlzI jstack: stack.txt

Server Log

Too big to upload - mostly useless

Ynverxe commented 1 week ago

Did you try using PaperMC?

EDIT: Try again with Pufferfish by removing the flags from the server startup script.

Ynverxe commented 2 days ago

@Jaren8r Hi! Your problem should be solved in version 1.3.8.

AdvancedPlugins / Seasons