Count one chunk per time, keep memory usage constant.

[Fenixin]

This pull request changes the way the blocks are counted, so it keeps RAM usage constant. I coded it in my way (I'm not really a programmer), hope you find it useful, but feel free to close the pull request if you don't like it.

With this pull request the list for the counts are stored in a dictionary and mian counts each chunk every time the chunk is opened.

[l0b0]

Awesome! I checked on a 4.1 MB world - This reduces the memory footprint by about 85%! Unfortunately the time to generate the plot went up by about 40%; do you have an opportunity to check whether this can be brought down to the earlier level?

One small thing I would undo is the conversion of bt_hexes to a set - The idea of keeping it a list was to get the same ordering in the output as the user specified on the command line. To get the best of both, it would be nice if main() simply removed any duplicates just before calling mian().

[Fenixin]

Ooops... just closed it accidentally... reopening. I will look into both things. Glad to see you like it!

[Fenixin]

This new commit uses the enumerate en the bt_hexes list (yep, its a good idea to keep the given order). Also it removes duplicates in main()... and, I accidentally added some code to print the status of the scanning process, it's quite basic, but works. That code was for another commit... but I mixed it!

About the slowdown, no idea at the moment... Have you any idea?

[pepijndevos]

I experimented with this a lot in Clomian. Some options: https://groups.google.com/group/clojure/browse_thread/thread/3c759739c265d14d/

Things to consider:

• Do I flip the chunk into layers? Alternative: use modulus and iterate
• Do I count per chunk, and sum up? Alternative: count on a global list
• How do I store things?

I'm sure there's more, but this is what I remember.

[l0b0]

OK, pulled. I would like to wait a few days to release a new version though, in case any of us come up with a patch to get the run time back to the previous level.

Fenixin: Thank you very much! The memory use really has gone down a lot now.

pepijndevos: The biggest performance gain from the first version was to eliminate the flipping - I think a single pass without any modifications to the data structure (but initializing it like Fenixin introduced) should be the fastest. I guess the devil is in the details, as usual when handling tons of I/O in a high-level language.

I'm looking forward to hearing more from you!

[pepijndevos]

I haven't touched this code in ages, but it looks to me as if we're looping over a layer for every block type. I think this could be done more efficient.

https://github.com/l0b0/mian/blob/master/mian/mian.py#L204

[Fenixin]

I think is better to wait a few days. Seems that the file save format of Minecraft is going to change in the next update:

http://twitter.com/#!/jeb_/status/33465975114305536

Here is the mod with documentation:

http://www.minecraftforum.net/viewtopic.php?f=25&t=120160

Better no waste time trying to optimize something that is going to change completely.

[pepijndevos]

Hm, maybe this is a good moment to write a NBT/area Protobuf.

[l0b0]

Wow, thanks for the info. I'll start a new branch for that so we can completely forget about the old file organization if Minecraft somehow converts games automagically.

[Fenixin]

@l0b0, I haven't the links now, but I'm pretty sure that Minecraft is going to convert the save games automatically (server, and cliente). Also, in the first post of the forum thread is a conversion tool.

@pepijndevos, the protobuf idea sounds really good, but I don't know if I can help with that, my programmers skills are limited at the moment (but I can try!).

[Fenixin]

More info:

http://mojang.com/2011/02/16/minecraft-save-file-format-in-beta-1-3/