Memory issues?

[brownan]

With the recent re-structuring of the program from recursive to iterative, the memory requirements went up a bit. I don't know how much exactly, it may have been negligible, but I wanted to open this issue to get feedback on it.

I have a couple ideas to improve memory efficiency, and I can also change it so that otherwise empty directories are not created saving disk space. But if none of this is a problem, then I'd rather focus on other things.

alexjurkiewicz, you were going to do a test on memory? Should I make improving memory usage a priority?

[eminence]

from what i've seen, memory usage seems to be very reasonable (for example, currently two process, each with 85M total memory, only 14M resident, during the tiling phase)

[brownan]

Oh good. I have some ideas in case memory does turn out to be an issue for even larger maps, but it sounds like it's not a big issue for yours, which has 9 zoom levels or 4^9 tiles at the highest level (minus blank tiles).

If I forgot to mention, I already did make a change so it doesn't create empty directories. That may save on disk space.

[SirPropane]

Running it on a 350MBish map, it is currently using 840M of RAM during the quadtree phase. For bigger maps it might be a good idea to look into the idea of sacrificing speed for RAM (if possible) as it is impossible to have a server and this running.

[brownan]

While I don't have any maps quite that big to test this, I would guess mosts of the memory is used by the processing queue. It is filled with a list of all the tiles at the highest zoom level and then slowly processed by the workers.

If your map has 9 zoom levels, that's 4^9 tiles. I would guess your zoom level even more than that, each new zoom level has 4 times as many tiles at the highest level.

The solution I have in mind is to only add the tiles to the queue, say, 10,000 at a time. That way there's no gigantic queue of every tile in memory at once. Unless I'm forgetting something, this is the only place the amount memory used is dependent on the input size.

[alexjurkiewicz]

We have 10 zoom levels now. The memory usage has gone too high for me to run on the primary server. I'll spin up temporary ec2 instances and run it there until there's a fix. This is what the process looked like after it rendered every tile (then I killed it): root 3781 49.8 49.7 803860 770308 ? RNl 11:46 7:40 python gmap.py -p 1 ../world/ /qt3minecraft.project357.com/map

I like your solution. It's the same concept rsync added in v3 for similar reasons. rsync adds 1050(?) items and whenever the number of tiles in the queue drops below 1000 they add enough to get back to 1050. That's a bit of a hacky solution, but it's quite effective. You could probably do with 100*numprocs or so.

[alexjurkiewicz]

Wow, we cracked the million tile barrier! I would expect less than 10% of those tiles to exist though :-)

[brownan]

As soon as I have some time free from work and school, I'll be making the change I mentioned earlier. I think I can easily cut the memory usage from being dependent on the input size.

[gshort]

Just imagine Google's tileset...18 zoom levels, and then all of that for each of their different layers, including satellite imagery, road overlays, the regular map, shaded relief topo maps...the list goes on. :D

[alexjurkiewicz]

Don't forget the maximum surface area of a Minecraft world is around three times the surface area of the earth...

[gshort]

Also, we render at something like 10 times the resolution google does, assuming 1 block = 1 meter (google's best imagery is, I think, maybe somewhere around 1 pixel per .5 meters, where as we're doing 16 pixels per meter.) All that in mind, I think it would take something like 25 zoom levels to render a complete Minecraft world. That's a rough estimate, though.

[eminence]

brownan, would having a copy of my large(ish) map be useful to you in testing? If so, I can make it available to you as a zip archive. Let me know

[brownan]

For the record, there are two places where memory is dependent on world size. The first is easy: all tiles for each level are shoved into the multiprocessing pool's queue at once. Solution: push something like 10,000 at a time and wait for them to complete.

The other is the chunks. A dictionary mapping chunk locations to chunk files is built and used during the tile processing. That'd be a little harder to factor out, but I don't think it's really too big of a problem since chunk counts are usually quite a bit smaller than the number of tiles.

[brownan]

Okay, so I've committed 32b776e6cc2f885b684285b995b07d8e33782e35 on a testing branch for now.

This change will only add 10,000 tasks to the process pool at a time, at which point it will wait for the queue to drain down to 500 tasks, when it will start adding tasks again.

This should help with the memory problems from very large maps. No longer are all 4^9 or 4^10 tile tasks going to be in some queue in memory at once.

I'll merge that into master tomorrow after I do a bit more testing to make sure nothing's broken. It's probably fine though.

[alexjurkiewicz]

Cool, I'll start a new map generation now!

[alexjurkiewicz]

Sorry, it still uses a heck of a lot of RAM. Here it is in the final 10 minutes of its processing (format is vsz rsz command): 1198024 1006808 python gmap.py -p4 --cachedir=cachedir ../world/ /snip/map/

That was me generating our map from scratch. If you want, I can add some debugging code to pickle the MP queue and chunk map to disk which should give rough ideas of memory requirements. Would that help?

[brownan]

First, how big is your map, in chunks and what is the detected zoom level? (Edit: You said earlier the zoom level is now 10. Still, how many chunks is your world?)

Also, you're using the code from the testing branch, right?

And finally, what would help me the most is to have a large map at my disposal to try things out with myself. The largest map I have on my computer is one where I hopped in a boat and sailed around for a while, but it's only around 6000 chunks.

[alexjurkiewicz]

I think I misunderstood git, I ran: git reset --hard 32b766e Upon actually checking the code it didn't do what I want >_> Once I figure out how to bend git to my will I'll retest. In the meantime here is a daily-updated link to our map. It's 10 zoom levels now. http://qt3minecraft.project357.com/qt3-minecraft-world.zip

[brownan]

Yeah, while git is powerful, in the sense that you can do most anything you can think of, there's no clear "right" way to do some tasks.

Assuming you have your own github repo and your local machine's repo, I would add my github repo as a remote with "github remote add brownan git://github.com/brownan/Minecraft-Overviewer.git".

Then fetch all new remote commits with "git fetch --all"

Then on your master branch, you can merge in my changes with "git merge brownan/master".

Things may need to work differently depending on your setup though. A reset should have worked provided that commit is in your repository... you may have just needed to "fetch" it first.

Edit: downloading your map now. Also, gitk is your friend.

[brownan]

Your world took longer to unzip than it did to download =)

Also, you should set the compression rate to 0 when you zip it. All the data files are already gzip compressed, so further compression doesn't help much and takes lots of cpu resources to compress and uncompress.

It looks like the compression ratio is 1.08, 753 MB went down to 699.

Thanks though, this will help me stress test things.

[mattrl]

Would just like to say that this has done a great job.

PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20 0 125m 49m 1580 R 95 1.3 9:24.44 python 20 0 125m 49m 1572 R 89 1.3 9:22.96 python 20 0 125m 49m 1572 D 99 1.3 9:23.71 python

Before I was using way too much memory I almost couldn't run overviewer with the server running.. or would have to drop the players to 15 and memory to 1.5GB.. but this is great.

Nice work.