Extreme memory usage during POIgen

[FastLizard4]

I'm experiencing extreme RAM usage during POI generation. Here's some sample output from a script that examines my system swap:

Overall swap used: 10315024 kB
========================================
kB      pid     name
========================================
1178592 18536   overviewer.py
1109352 20855   overviewer.py
1108252 20852   overviewer.py
1080948 20851   overviewer.py
1072716 20850   overviewer.py
1071996 20853   overviewer.py
1071620 20854   overviewer.py
356480  18829   overviewer.py

I can safely say that this is more memory usage than four CraftBukkit servers running simultaneously, and is even considerably larger than the total size of the Minecraft world these guys are processing:

minecraft@ridley:~$ du -hs s2/Bob/region
1.8G    s2/Bob/region

Although I do not recall exactly when I started seeing this, it is something recent (within the last two months). Here is my version information, note that I get Overviewer from the APT repository:

fastlizard4@phazon:~$ overviewer.py --version
Minecraft Overviewer 0.11.90 (a278019)
built on Mon Jan  6 08:46:16 2014

I can't even begin to afford a server with enough memory to satiate POI gen, and it causes the server that does run it to swap itself to death every time it runs. At this point, even a small reduction in RAM usage during POI gen would be appreciated!

[CounterPillow]

I guess every worker has its own copy of the world in the memory, which is certainly not desired. Try reducing the processes count for genpoi. (If you can even do that separately from the command line. Not sure if it just works globally for both render and genpoi through the config file.)

[FastLizard4]

Reducing the process count for poigen to 1 does help, but it's still demanding more memory than my server even has.

Additionally, although I cannot currently give an exact figure, I can say that the single Overviewer process doing poigen is using more memory than my total world size.

[CounterPillow]

I believe that is because minecraft worlds are gzipped, and also there's python's overhead.

[FastLizard4]

A six-process mapgen uses barely a third of the memory a one-process poigen uses. And why would Overviewer need to hold the entire map in memory to do a poigen? It clearly doesn't need to for mapgen, and the idea that it's necessary because the Anvil files are gzipped is a fairly absurd reason. The Minecraft server itself is very clearly capable of dynamically loading and unloading the map as needed, despite this.

[wlritchi]

While there is certainly overhead from Python as well as increased size from unzipping the chunk data, that doesn't begin to explain why Overviewer should need to hold the entire map in memory. Frankly, it should never need more than one chunk in memory per thread. What's more, it doesn't have any real reason to store any chunks in memory; since NBT is a stream format, it can simply read the stream directly and ignore everything except for the tags relevant to the POIs.

[FastLizard4]

Here's a graph I pulled from my Munin installation and annotated in Photoshop to give a visual representation of what happens when Overviewer runs.

[CounterPillow]

I never said that Overviewer needs to hold the entire map in memory. That is clearly a bug. No need to get all aggressive. I'm simply trying to explain the symptoms. This will be fixed when someone gets the time and motivation to fix it, or as always with opensource projects, you're able to fix this yourself if "when someone else does it" isn't soon enough for you.

[PIBM]

My original speed up (multithread) of gen poi didn't require to load more data or more memory in itself. It was already requiring 48GB to render my 25gb world.

I did my newest fix to the genpoi so that it would only load the modified tiles mostly to get rid of this issue as I was running out of memory on my 25GB world server and had to render it on a server with 128GB of ram.

The reason so much memory is allocated is that all of the entities are held live until they are gathered, then they are filtered and written to disk. My new mod only load the entities of the tiles that have been modified, load all the previous poi generated and remove any poi from tiles that have been rescanned, then merge both. Sadly, this will only help you for re-render.

I will look into filtering the entities as they are read so that much less information needs to be kept until the end.