Querz / mcaselector

A tool to select chunks from Minecraft worlds for deletion or export.
MIT License
3.25k stars 178 forks source link

Default process threads count causes out of memory errors with high core CPU #328

Closed DefectiveNu closed 2 years ago

DefectiveNu commented 2 years ago

when using a high core count cpu (32 in my case) the default values for process-threads (30) causes out of memory errors which stalls processing at some point

To Reproduce Steps to reproduce the behavior:

  1. run commandline: MCA Selector.exe --mode delete --region world\region --query "InhabitedTime = 0"
  2. log should show some threads reporting "java.lang.OutOfMemoryError: Java heap space"
  3. progress will stall before completion (assuming its waiting for the crashed threads)

Expected behavior process eventually finishes

Screenshots and other files CLI output seems a bit garbled but here's the relevant segment:

Exception in thread "processPool-thread-13" java.lang.OutOfMemoryError: Java heap space
        at java.base/java.io.BufferedInputStream.<init>(Unknown Source)
        at java.base/java.io.BufferedInputStream.<init>(Unknown Source)
        at net.querz.mcaselector.io.mca.Chunk.load(Chunk.java:47)
        at net.querz.mcaselector.io.mca.MCAFile.load(MCAFile.java:252)
        at net.querz.mcaselector.io.mca.Region.loadRegion(Region.java:111)
        at net.querz.mcaselector.io.mca.Region.loadRegion(Region.java:32)
        at net.querz.mcaselector.io.job.ChunkFilterDeleter$MCADeleteFilterProcessJob.execute(ChunkFilterDeleter.java:78)
        at net.querz.mcaselector.io.job.ProcessDataJob.run(ProcessDataJob.java:73)
        at net.querz.mcaselector.io.JobHandler$WrapperJob.run(JobHandler.java:237)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Exception in thread "processPool-thread-5" [03:01:05.112] failed to load any data from java.lang.OutOfMemoryError: Java heap space
r.-15.9.mca
88.56%  r.-15.9.mca
[03:01:05.427] failed to load any data from Exception in thread "processPool-thread-32" r.-15.10.mcajava.lang.OutOfMemoryError: Java heap space

88.86%  r.-15.10.mca

note: its not always at the same file

Querz commented 2 years ago

Added a hard limit of 4 threads in 1.17.3. This can still be increased manually if needed.