Out of memory for 100+ GB map

ShDis commented 8 months ago

Crashed after ~15 minutes of collecting data (16GB RAM, Purpur 1.20.2)

! [пт 18:40:05 WARN  BlueMapSignExtractor] Plugin BlueMapSignExtractor v1.2 generated an exception while executing task 67
java.lang.OutOfMemoryError
    at java.base/java.util.zip.Inflater.inflateBytesBytes(Native Method)
    at java.base/java.util.zip.Inflater.inflate(Inflater.java:378)
    at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
    at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:122)
    at java.base/java.io.FilterInputStream.read(FilterInputStream.java:82)
    at BlueMapSignExtractor-1.2.jar//de.bluecolored.bluenbt.NBTReader.readTag(NBTReader.java:462)
    at BlueMapSignExtractor-1.2.jar//de.bluecolored.bluenbt.NBTReader.peek(NBTReader.java:57)
    at BlueMapSignExtractor-1.2.jar//de.bluecolored.bluenbt.NBTReader.checkState(NBTReader.java:500)
    at BlueMapSignExtractor-1.2.jar//de.bluecolored.bluenbt.NBTReader.beginCompound(NBTReader.java:77)
    at BlueMapSignExtractor-1.2.jar//de.bluecolored.bluenbt.adapter.DefaultAdapterFactory$DefaultAdapter.read(DefaultAdapterFactory.java:129)
    at BlueMapSignExtractor-1.2.jar//de.bluecolored.bluenbt.BlueNBT.read(BlueNBT.java:109)
    at BlueMapSignExtractor-1.2.jar//de.bluecolored.bluenbt.BlueNBT.read(BlueNBT.java:117)
    at BlueMapSignExtractor-1.2.jar//com.technicjelle.bluemapsignextractor.common.MCA.getBlockEntities(MCA.java:51)
    at BlueMapSignExtractor-1.2.jar//com.technicjelle.bluemapsignextractor.common.Core.processMCA(Core.java:37)
    at BlueMapSignExtractor-1.2.jar//com.technicjelle.bluemapsignextractor.common.Core.lambda$loadMarkers$1(Core.java:23)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
    at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
    at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
    at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Ab

But it's sounds like very cool plugin for me, many thanks to you by advance! (Actually have been looking for a long time to be able to display signs on the map)

TechnicJelle commented 8 months ago

Thanks for the bug report!

I've been hard at work this evening with refactoring the plugin more. There are still a couple of bugs, but if you're willing to try it out already, I'll attach a preview version that you can use.

If you do try it, please let me know if it fixes the issue! It seems to at least already be much faster, if anything.

BlueMapSignExtractor-1.3-PREVIEW.jar.zip

ShDis commented 8 months ago

With 1.3-PREVIEW.jar

[14:48:18] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loading textures...
[14:48:22] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Baking resources...
[14:48:22] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Resources loaded.
[14:48:22] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loading world '97a8f878-4494-43d1-950e-9178e5d0e1d6' (C:\2s2d\world)...
[14:48:22] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Initializing Storage: 'file' (Type: FILE)
[14:48:22] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loading map 'world'...
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loading world '314ebe7f-8190-424a-8fb8-7c5f8324dba3' (C:\2s2d\world_the_end\DIM1)...
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loading map 'world_the_end'...
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/WARN]: [BlueMapSignExtractor] New version available: v1.2 (current: v1.3-PREVIEW)
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/WARN]: [BlueMapSignExtractor] Download it at https://github.com/TechnicJelle/BlueMapSignExtractor/releases/latest
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMapAreaControl] BlueMapAreaControl enabled!
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMapAreaControl] Loading existing configs
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMapAreaControl] Loading config for map: world
[14:48:23] [Craft Scheduler Thread - 3 - BlueMapSignExtractor/INFO]: [BlueMapSignExtractor] World "world": Extracting signs into markers...
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMapAreaControl] Loading config for map: world_the_end
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loaded!
[14:49:05] [Craft Scheduler Thread - 3 - BlueMapSignExtractor/ERROR]: [BlueMapSignExtractor] Error reading region file
java.io.EOFException: null
    at com.technicjelle.bluemapsignextractor.common.MCA.readFully(MCA.java:153) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.MCA.readFully(MCA.java:141) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.MCA.loadChunk(MCA.java:111) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.MCA.getBlockEntities(MCA.java:70) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.Core.fillMarkerSetFromRegionFile(Core.java:57) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.Core.lambda$loadMarkerSetFromWorld$2(Core.java:42) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
    at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845) ~[?:?]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
    at com.technicjelle.bluemapsignextractor.common.Core.loadMarkerSetFromWorld(Core.java:42) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.Core.addMarkersToBlueMapWorld(Core.java:24) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.BlueMapSignExtractor.loadMarkersFromWorlds(BlueMapSignExtractor.java:67) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.BlueMapSignExtractor.lambda$new$0(BlueMapSignExtractor.java:50) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at org.bukkit.craftbukkit.v1_20_R2.scheduler.CraftTask.run(CraftTask.java:101) ~[purpur-1.20.2.jar:git-Purpur-2095]
    at org.bukkit.craftbukkit.v1_20_R2.scheduler.CraftAsyncTask.run(CraftAsyncTask.java:57) ~[purpur-1.20.2.jar:git-Purpur-2095]
    at com.destroystokyo.paper.ServerSchedulerReportingWrapper.run(ServerSchedulerReportingWrapper.java:22) ~[purpur-1.20.2.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.lang.Thread.run(Thread.java:833) ~[?:?]
[14:49:05] [Craft Scheduler Thread - 3 - BlueMapSignExtractor/ERROR]: [BlueMapSignExtractor] Error reading region file
java.io.EOFException: null
    at com.technicjelle.bluemapsignextractor.common.MCA.readFully(MCA.java:153) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.MCA.readFully(MCA.java:141) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.MCA.loadChunk(MCA.java:111) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.MCA.getBlockEntities(MCA.java:70) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.Core.fillMarkerSetFromRegionFile(Core.java:57) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.Core.lambda$loadMarkerSetFromWorld$2(Core.java:42) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
    at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845) ~[?:?]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
    at com.technicjelle.bluemapsignextractor.common.Core.loadMarkerSetFromWorld(Core.java:42) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.common.Core.addMarkersToBlueMapWorld(Core.java:24) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.BlueMapSignExtractor.loadMarkersFromWorlds(BlueMapSignExtractor.java:67) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at com.technicjelle.bluemapsignextractor.BlueMapSignExtractor.lambda$new$0(BlueMapSignExtractor.java:50) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
    at org.bukkit.craftbukkit.v1_20_R2.scheduler.CraftTask.run(CraftTask.java:101) ~[purpur-1.20.2.jar:git-Purpur-2095]
    at org.bukkit.craftbukkit.v1_20_R2.scheduler.CraftAsyncTask.run(CraftAsyncTask.java:57) ~[purpur-1.20.2.jar:git-Purpur-2095]
    at com.destroystokyo.paper.ServerSchedulerReportingWrapper.run(ServerSchedulerReportingWrapper.java:22) ~[purpur-1.20.2.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.lang.Thread.run(Thread.java:833) ~[?:?]

TechnicJelle commented 8 months ago

Ah dang, I was hoping this was a rare error, but it seems like it's more common... Thanks for giving it a shot! I'll try to diagnose this issue further, but I couldn't really figure it out yesterday. Maybe today I'll have some fresh thoughts...

TechnicJelle commented 8 months ago

Thanks to Blue, I was able to solve the EOF Exception! Could you please try out this new version?

BlueMapSignExtractor-1.3-PREVIEW2.jar.zip

ShDis commented 8 months ago

Cool! It seems to be working. I think it'll take some time to complete. Very curious to see the result (last time I collected the signs with Overviewer and with one working core of E8600 it took about three days). The map also contains some old chunks from around 1.12 or earlier

TechnicJelle commented 8 months ago

Ah, I'm glad to hear that it's working now!

And wow, that's a long time and a big world, then! I feel like I should mention that this plugin extracts the signs from all your world files every time BlueMap loads, which is at least every server startup... (But also when you do /bluemap reload) I have been hoping that won't turn out to be a problem, but with worlds as large as this, it probably will...

And this plugin doesn't support chunks older than 1.13. I am pretty sure it'll just skip the old chunks, though; it shouldn't stop the whole process. But please open another bug report if it does end up stopping the whole process.

TechnicJelle / BlueMapSignExtractor

Out of memory for 100+ GB map #34