New cache format

[eminence]

One of the reasons minecraft shifted from the old chunk format to the new region format was to reduce the number of small files that Minecraft had to open and read. Since Overviewer still does the same thing (creating a cache image for each chunk), we too sould see a speed boost if we changed this. I think we have two options:

• Bigger chunk cache images. something bigger than we use now (1 chunk per image). We could probably go all the way up to 1024 chunks (32x32, the same size as regions), but this may be too large
• Get rid of the cache entirely, and render directly to tiles.

[Xon]

The profiling data had; ImageDraw.draw was ~6% total runtime, (452k calls) ImageDraw.line ~0.97% (65k calls) alpha_over's 8% (910k calls)

~13% of the render function is in loading chunks. moving to a FILO cache should dramatically reduce that.

Just moving "draw = ImageDraw.Draw(img)" from; if blockid not in transparent_blocks or blockid in (78,): #special case snow so the outline is still drawn
draw = ImageDraw.Draw(img) if x != 15 and blocks[x+1,y,z] == 0: draw.line(((imgx+12,imgy+increment), (imgx+22,imgy+5+increment)), fill=(0,0,0), width=1) if y != 0 and blocks[x,y-1,z] == 0: draw.line(((imgx,imgy+6+increment), (imgx+12,imgy+increment)), fill=(0,0,0), width=1) To; if blockid not in transparent_blocks or blockid in (78,): #special case snow so the outline is still drawn
if x != 15 and blocks[x+1,y,z] == 0: draw = ImageDraw.Draw(img) draw.line(((imgx+12,imgy+increment), (imgx+22,imgy+5+increment)), fill=(0,0,0), width=1) if y != 0 and blocks[x,y-1,z] == 0: draw = ImageDraw.Draw(img) draw.line(((imgx,imgy+6+increment), (imgx+12,imgy+increment)), fill=(0,0,0), width=1) Should save ~387k calls or a drop about 5% of the total runtime off. http://xonwaix.shacknet.nu/images/gmap6.png

[agrif]

eminence, I'd like to go over your code and double-check the increfs/decrefs, since you noted this in the comments. While I'm there, I'll rearrange things to be C89 compliant; especially on windows, not everyone is fortunate enough to use GCC.

I think we should have a contrib/ script to clean out old caches from the worlddir, since we're not using it anymore. But that's not really something that has to get done right away.

Also this will be a pretty huge change when it's merged, but anyone who doesn't check github probably won't be able to tell. Maybe we could take this opportunity to rename gmap.py to something more suitable, like overviewer.py? This would certainly be an indication that something big has changed, but it could also cause user confusion.

As for --zoom, that's always been handled in quadtree.py. As far as I know, it still works just fine: it tells Overviewer how big to make the map, instead of having it figure this out on its own from min/max chunk coordinates. brownan wrote in the README that this isn't a particularly good name; maybe now would be a good time to change it.

[slowriot]

I'm looking forward to seeing this rolled out.

Meanwhile, if anyone would like to perform local tests with a large world (1.5GB without biome data or cache), help yourself to the latest backup tarball at: http://minecraftonline.com/backups/

Hope that's of help to someone!

[Fenixin]

Just testing the dtt-c-render branch with a huge world (thanks Marcel!) in a quad core machine. Everything is working as expected, but for this sizes of map there is a lot of tiles to skip, and is expending almost the same amount of time skipping tiles as rendering. Look this image with the use of the four cores:

http://imgur.com/hGj8c

When they are almost idle they are skipping tiles (time scale in seconds). From the log info it skips 1000 tiles in ~1.75 seconds. Is still running the first level, so no info about gow is skip process in > 1 levels.

If nobody does it first, I will run a profiling of the skipping process.

[edit] the map needs 11 levels of zoom by default

[eminence]

Fenixin, that's very interesting. I looks like at those points Overviewer is disk-IO-bound (and not CPU-bound like it is while rendering) profiling data will be helpful to see the bottlenecks.

also, if any of you are on IRC, feel free to join #overviewer on freenode. it may be easier to have conversations there in real-time, rather than on github.

[Fenixin]

With the second tiling level it skips 1000 tiles in ~0.3 seconds (sometimes more, sometimes less). So that's much faster.

[Fenixin]

Here is the profiling file and the png call graph for 100.000 skipped tiles:

file: http://www.mediafire.com/?poy3z6nw873kz9c

png call graph: http://imgur.com/NKZyy

Seem like the os.path.exists in quadtree.py line 441 is the problem.

[Xon]

Have a look at the master branch in my fork, this commit; Both os.path.exists and os.path.getmtime calls os.stat. makes a fairly major difference in the update scan process.

[Fenixin]

Works great Xon, but only single threaded, using 2 or more cores slow downs the scan a lot, didn't check the code... is that expected?

[Xon]

Just going to distil the mtime check changes and the region path lookup into another branch for sending a pull request. The master branch has some poorly optimized junk in it while I'm trying to unbreak.

[Xon]

Ok, my dtt-c-render-lookup branch is better now. There appears to be a performance regression during update scanning with more than 1 worker process, but single worker performance is great.

:edit:Just adding debug print statement in render_worldtile shows it just isn't dispatching tiles to the worker process's efficiently. I'm also profiling the parent process, but I expect it is sitting a lot of time in acquire for pushing data across the shared queue.

:edit: I've added batch job support to the multiprocess queuing. I'm not sure if the stalls which it's encountering are caused by adding new items or pulling off the results or the reporting being funky

[Xon]

Hopefully got a fix for the massive performance impact on multi-workers. Just need to test some code on Windows to ensure it works correctly.

Yup fixed it. Existing bug which my region caching code just made worse.

[TravisArmstrong]

I just want to say keep up the great work guys, your hard work is appreciated!

[agrif]

I just ported the get_lighting_coefficient function to C, which was the last major bit of lighting code still written in python. This led to a major speedup, as noted in the table here. I also ported over the spawn render mode, and fixed up transparent block lighting.

Since all the development is currently going on in the dtt-c-render branch (and its children), I deleted the older direct-to-tiles branch. Also, since the lighting code now seems fairly complete, I merged that in and deleted those branches. If anyone has any particular need for those branches to exist, feel free to tell me and I'll remake them. I was just getting tired of having them cluttering up my repo. :P

[Xon]

Multi-layer rendering looks to be in a working state. Just requires some cleanup agrif's googlemap.py to remove some of the restrictions and wire it up to the commandline/config file options.

I'm going to hammer at my branch with some rebase & cherry-pick so the pull request isn't so evil.

[cheeplusplus]

These speedups look amazing. My map (301MB, 86 regions) took several hours to render from scratch even with four cores, after switching to MCRegion, and has been taking longer than before to do periodic updates since we lost the chunklist. Can't wait to see this all put together.

[cjdaniel]

I just tested the dtt-c-render branch, and I am also impressed with the speed improvement. My 105MB, 43 region map took about 20 minutes to render from scratch, and then performed an update in 13 seconds. I, too, am excited to see this polished and merged into the main branch! Thanks for all your hard work!

[eminence]

Unless there are any objections, I would like to propose the following plan:

This weekend we will push the latest code from agrif's fork into a new branch of brownan's repository. We will provide updated documentation and Windows binaries for people to test with. Any bugs found in this version will be fixed. Any new feature requests will be done in another branch or deferred.

After about a week, assuming that there are no outstanding critical bugs, we will merge this new code into brownan's master branch (while saving the current master in a new branch).

[Xon]

Sounds good, I want to finish integrating the biome caching into the existing region & chunk caching and tweak the cache invalidation logic.

Probably will want to merge all those branches on agrif's repos into a single one and tets it before pushing to brownan's repos

[Fenixin]

For me there's still one missing graphical feature, the edge block line, I think it improves the visualization of the maps a lot.

About them I was thinking in using different colors for different blocks, instead of constant black. My idea was to add to special_blockmap and blockmap a tuple with the color of the line (so both blockmaps give a tuple for every blockid, the first term is the texture and the second one is the tuple with the color). This color can be hard coded, but to make it texture pack dependant, we can take the average color of the top texture of the block (or maybe the side? needs some experimentation). What do you think? If you like the idea I'll code it during Friday. (probably)

And I agree with Xon, better to merge all those branches on agrif's repo.

[eminence]

i think that would create much better looking stuff. my though was to actually make the lines semi-transparent, instead of black. i think that would have roughly the same effect as your idea.

to your first point, yes, the block edge lines are an important missing feature!

[agrif]

While we go through and update the various documentation, it'd probably be nice to provide guides to

• people who have used Overviewer, but are upgrading, to explain the big differences and new features, and
• people who have never used Overviewer before and want to get started.

Looking around online it seems that all the feedback for Overviewer is either "wow, it's awesome!" or "I don't know how to use it :(". It even seems a lot of people think you need to know python or javascript to use it! This is part of a bigger, general problem with the Overviewer docs, but a few "getting started" guides would probably help out a lot.

[eminence]

We have just pushed the dtt-c-render code from agrif's fork into brownan's fork. In time, this code will be merged into master.

This new direct-to-tile (or DTT) version has a number of big changes:

1. It no longer uses a chunk cache (so the --cachedir option has been removed)
2. Building the C-extension is now required. Binary kits are provided for Windows users. Other platforms will need to compile this extension manually, but often this is as simple as running python setup.py build
3. It is much faster! Up to six times faster!

It should be noted that many of the command line options have changed, so please look at the README or run --help to get a summary of the new options.

We will be writing some documentation in the next few days on this new version to put in our wiki and README (how to run it, build the C-extension, etc). In the meantime, if you want to try this new code out but are having problems, please join #overviewer on freenode and we'll help you out. Please don't reply to this issue, since it's getting rather lengthly and hard to follow.

Finally, a big thank you to everyone who had contributed and helped test this new version. We've made 183 commits, and touched nearly every single file over the past month, and we think that we've come up with a great set of improvements.

Thanks!

[duckman]

I'm glad the C-extension is required now. Mine randomly decides to not work (probably do to a system update, im on gentoo so I break things alot :-P) and so I have like half my chunks with those ugly lines in the water :-P.

[bjorhn]

Does rendering multiple modes using settings.py not work yet? I have this set: rendermode=["lighting"] rendermode.append("night")

But it doesn't work. I also tried rendermode=["lighting,night"] since I'm not too familiar with the Python syntax and I'm mostly just guessing here, but that doesn't work either. Should I just use the command line for now? Everything else works well, awesome job!

Edit: The same seems to go for procs. I have procs = 1 set in settings.py but overviewer is using both available cores for tile rendering. Am I doing something wrong with my settings file?

[BarryCarlyon]

rendermode=['lighting", "night"]

not sure about procs, I only have one processor in mine

the same settings file is new I am told so some of it might not work

[eminence]

using rendermode.append and procs should work. you can use the --display-config option to get overviewer to print out what config settings it will use:

$ cat settings.py 
rendermode=["lighting"]
rendermode.append("night")

procs = 1

$ python overviewer.py ../../minecraft/dev_world output_dir --display-config
2011-04-08 07:44:35,526 [INFO] Using the following settings:
version: None
procs: 1
zoom: None
delete: None
chunklist: None
rendermode: ['lighting', 'night']
list_rendermodes: None
imgformat: None
optimizeimg: None
web_assets_hook: None
quiet: 0
verbose: 0
skipjs: None
display_config: True

[bjorhn]

Weird. If I use --display-config it does display everything correctly, both the procs and the rendermode. However, when I run overviewer.py, it still only renders one layer, and it still spawns two instances that each use 100% of my two cores.

2011-04-08 13:53:16,442 [INFO] Using the following settings: version: None procs: 1 zoom: None delete: None chunklist: None rendermode: ['lighting', 'night'] list_rendermodes: None imgformat: None optimizeimg: 1 web_assets_hook: None quiet: 0 verbose: 0 skipjs: None display_config: True

[electrofloat]

Hey, I've just tried out the dtt-c-render branch and it is super fast indeed. Though there's one bit of a problem (but this could be on my side) the redstone wires not always appear right. Like on this picture: http://img818.imageshack.us/i/redstonewire.png/ Those red and dark red blocks should be redstone wires.

[Fenixin]

Can you please try to update your terrain.png? Looks like you are using the old one, redstone texture changed his position in the terrain.png in one of the last updates.

[electrofloat]

Ohh crap. I knew that this should be a problem on my side. Thanks, updated and it works flawlessly. (The confusion came from that I installed the minecraft.jar in ~/.minecraft/bin though there was a terrain.png also in Minecrafto-Overvier's directory. So it used the old terrain.png file)

[agrif]

dtt-c-render has now been merged, so I'm closing this.