Add support for halloween update.

[alexjurkiewicz]

It looks like:

• Both layers get rendered. Surface chunks appear normally but if there is no surface chunk for a location the hell chunk will be used instead.
• Tile colours messed up
• Hell has an adminium roof that needs to be hidden.
• More??

[alexjurkiewicz]

I'm looking into the new chunk storage format now. I'm thinking the best solution is to add a --hell trigger to render hell, and we can worry about fixing hell rendering and unifying the layers in the other ticket.

[alexjurkiewicz]

OK, the reason grass is all grey now is because the texture for grass is now greyscale. A tint is getting applied later on, I guess, so each Biome is a different colour. No other tile is being treated like this.

There are 7 new textures: four for the pumpkin (front (unlit), front (lit), sides, top) and three for the new hell blocks (hellstone, mud, hellglass).

[alexjurkiewicz]

Oh, leaves have been greyscaled too.

[rbos]

maybe there could be a layer that could replace the current layer, scaled to 16x.

That way you could flip between them and get equivalent locations between the worlds.

[brownan]

working on this now

[brownan]

Get version 32552c50d643a308d3b2024ad9360e4f2ba3c0cc, a one line change to only render the overworld.

Still to do: get textures rendering with the proper colors, which I'll have to apply myself. Hope grayscale is okay with everyone for now.

[alexjurkiewicz]

There is a grass.png in the client jarfile that has green grass. Perhaps it's worth sourcing that for grass's sprites until a more permanent fix is in place.

@rbos: shouldn't it be possible to use js magic to define a multiplier offset between layers? So if you're looking at 0,0 on the surface and change to The Nether, you see 0,0. But if you're looking at 1,1, you'd see 16,16 instead.

[alexjurkiewicz]

Here is a quick patch to use green grass rather than grey grass until something more permanent is put in place. Sidenote: It took me ages to figure out how all the magic in chunk,py works. Thanks for taking the time to figure it out in the first place :).

diff --git a/textures.py b/textures.py
index b942168..0a779b9 100644
--- a/textures.py
+++ b/textures.py
@@ -333,6 +333,16 @@ def load_water():
     lavablock = _build_block(lavatexture, lavatexture)
     blockmap[10] = lavablock.convert("RGB"), lavablock
     blockmap[11] = blockmap[10]
+    
+    # Temporary workaround
+    # The grass texture in terrain.png is greyscale, and is tinted by
+    # Minecraft dynamically.
+    # This overrides that texture with a pre-tinted one, until proper biome
+    # support is added.
+    grasstexture = _load_image("grass.png")
+    grasstexture = grasstexture.convert("RGBA")
+    grassblock = _build_block(grasstexture, terrain_images[3])
+    blockmap[2] = grassblock.convert("RGB"), grassblock
 load_water()

[eminence]

does anyone know if the tint of the grass/leaves is stored in the game files (perhaps in the Entity structure)?

i can take care of adding the new textures (pumpkins, hellmud, etc) in the next day or so, if no one gets to it first

[agrif]

If I had to guess, grass/leaves tint would be in the block metadata (like torch orientation, ...). I haven't checked this though.

[alexjurkiewicz]

I have untested code to draw pumpkins, I'll post it tomorrow when I'm back at work.

[eminence]

I've pushed to my fork an updated textures.py that renders the new blocks:

eminence/minecraft-overviewer@2c49113bd2d731a15a90ccdfe1e89ff7b16b9dcb

I've tested it, but if someone else can give it a go before I submit a pull request, that would be appreciated. Grass is still grayscale. I'm tackling that next.

[eminence]

i've pushed another commit to my fork:

• addresses a small pumpkin problem that alexjurkiewicz pointed out
• adds a hacky tint to grass and leaves so they're no-longer grayscale. the tintTexture function needs improvement.

eminence/minecraft-overviewer@290269f8fea445e6dfc769a59b0a0b5e5ebe33dd

note that i had an already outstanding pull request. it appears that it has automatically been updated to include these new commits. that's the magic of github, i guess :)

[alexjurkiewicz]

On pull requests: yeah, a pull request is branch-to-branch, so if you add more stuff to the branch it magically appears in the pull request.

[agrif]

I'm not sure if any of you have seen this, but after a very helpful response from the minecraft wiki, I found this page, that has information on how leaves and grass are tinted. Looks like you need to calculate a "moisture" and "temperature" value that isn't stored with the chunks (sigh).

Maybe we can talk Notch into caching the biome info? In the meantime, we can just sample the grasscolor.png and foliagecolor.png files at reasonable guesses to get texture-pack appropriate tinting.

[alexjurkiewicz]

Further down that thread someone posted code to find the temp/moist values: http://www.minecraftforum.net/viewtopic.php?f=25&t=71923#p1136670

It looks like they call Minecraft code directly to get the data. I guess we could do that too -- but it adds a pretty major dependency.

[eminence]

jython can run native java code, but the latest version of jython is only 2.5.2, so it may take some work to get [the current version] of overviewer working (pretty essential things like numpy and multiprocessing aren't working on an out-of-the-box jython installation).

another option would be for a java guru to reverse engineer the pb.class algorithms, but this is very non-trivial. from what I can see, the algorithm that calculates temp/moisture is only based on a single seed value (much like the terrain generator is only based on the seed value). this indicates to me that the algorithms involved are sufficiently complex that this option is just not feasible.

we should first fix up the tintTexture() function to produce less crappy results before we get all gung-ho about implementing true-to-game coloring

[that forum post is a great find, though, agrif. at least now we know how hard this might be :)]

[alexjurkiewicz]

Maybe ImageOps.colorize() is the method needed for good results? I'll leave the PIL mangling to you guys though :-)

[agrif]

If I were Notch, I'd implement the grass tinting with vertex colors. The default blending for vertex colors in OpenGL is straight up multiplication -- this makes a sort of sense, given that the grass texture is now grey. I would start with that. (this can be done with ImageOps.colorize(image, (0, 0, 0, 255), color), like alex suggested)

One of the suggestions in that thread was (I think) using an external, Java program to generate biome-maps of chunks that could be read in. That could work, but using Minecraft code itself makes Overviewer even harder to get working out of the box. Also, we'd have a mapper that uses 3 different source languages. I already have a hard enough time switching from Javascript to Python without throwing extra semicolons everywhere :D

Honestly, the best solution would be for Notch to put this info in the chunk file. He already caches light data, but the biome info has to be recalculated on the fly. That's kinda weird. It's not really a priority for him, though, I'm sure. Just wishful thinking on my part.

But yeah, certainly get the tinting approximately correct before we worry about this mess.

[eminence]

ok, so the colorize code i came up with looks like this:

 def tintTexture(im, c):
-    color_map = []
-    for component in c:
-        color_map.extend(int(component/255.0*i) for i in range(256))
-    return im.point(color_map)
+    # apparently converting to grayscale drops the alpha channel?
+    i = ImageOps.colorize(ImageOps.grayscale(im), (0,0,0), (170,255,50))
+    i.putalpha(im.split()[3]); # copy the alpha band back in. assuming RGBA
+    return i

I tweaked the tint color a bit. it's closer to the original grass color, but a bit darker. it instead of this, it now looks like this. there is something weird going on with the top of the leaves that i don't understand

[agrif]

That looks a lot better, like how it was before the update.

The blue tint to the leaves might be the old blending bug cropping up again, but it looks a little different than that. I dunno, shrug.

[brownan]

I've pulled in the changes by eminence. Thanks a lot for looking into that for everyone.

looks like proper biome colors are going to be tough. But at least maps aren't grayscale for now.

[agrif]

After getting a chance to try it today, I can confirm that the blue tint on the leaves is the blend bug. I've got a branch with your new leaf/grass code, but using alpha_over, and the blue tint is gone. I'll submit a tiny pull request soon, but first I'll fix some issues I'm getting compiling the extension on Mac OS X.

[Cryssy]

Any way to actually get it to render the new blocks?

Thanks, Cryssy

[eminence]

Cryssy, try using this new script I just committed to my fork:

eminence/Minecraft-Overviewer@d0267d4e7863e72f93dcb91ff7a3e199bb1eafbe

Please note that to use this, you'll also need the fix to --chunklist that's in my fork (this fixes a bug I accidentally introduced in my signpost handling code)

[eminence]

based on the link that agrif posted, i hacked together some java code that'll provide the necessary biome information to Overviewer, so grass and leaves can be tinted correctly. Here's the result:

http://smp.em32.net/biome_test/

(Note that due to my laziness the entire grass chunk is tinted, not just the top texture).

The render is stupidly slow and inefficient as each grass and leaf block is tinted on the fly (after making a round-trip out to a java process to figure out the correct moisture/temperature). the results, however, do look good.

[alexjurkiewicz]

Questions:

1. Does your code deal with cases where much of the world was generated pre-biome?
2. Your code seems to have smooth gradients between biomes rather than the binary transitions detailed by the code. I assume this is easy to fix?
3. Want. Code. :P

[agrif]

Something I just discovered...

There's a Biome Extractor over on the Minecraft forums, made specifically for mappers, but not tied too strongly with any one mapper. I think it was made originally for mcmap. That might be a good fit.

It looks like it handles chunk caching in a way similar to Overviewer, and using this tool instead of an Overviewer-specific tool means we can rely on external help when the biome algorithms are updated. Also, it'd be great for people who use more than one mapper, because the biome info will be cached for all of them.

[eminence]

Alex, since my code uses the class files from minecraft.jar:

1) Yes, in the same way that minecraft deals with this

2) The code described in the minecraftforums.net link takes a global block coordinate, and returns the biome information, so there was nothing for me to "fix"

3) I'll post it as soon as I can clean it up a bit (that way you won't look at it and think "wtf")

Agrif, that's very interesting. Doing a 1-time image read to get the color information for each block should be a lot faster than asking java at each step. I think we'd still have to tint on the fly, as I'm not sure that it's feasible to cache the 65k possible grass/leaf textures (though I'll give it a try to see how that goes)

[agrif]

Tinting on the fly should be reasonable. My lighting code does what basically amounts to 3 tints per block, and is only moderately slower. One tint per grass block may even be hard to notice. Besides, you can make biome usage optional with a command line flag.

Alex, while the biomes themselves are well-defined and binary, the grass and leaf colors are gradually changing. This may be the source of the confusion. If you want well-defined biome boundaries, you can use a colormap png with blocks of color instead of the default gradient.

[eminence]

I've pushed a version that reads biome data from the Biome Extractor data. The results look the same (here's another example: http://smp.em32.net/biome_test2/)

eminence/Minecraft-Overviewer@574ee7ad6ebb74edc550169b945f0406d36d1660

• You need to have run the Biome Extractor tool manually first. The biome pngs also need to be copied into your working directory
• Only the tops of grass is tinted (unlike the entire block in my earlier code). The bright-green part of the side texture is still pretty distracting, though. I would like to find an efficient way to fix this.
• Rendering is noticeably slower. If someone wants to take a stab at profiling and optimizing this code, that would be great.

[alexjurkiewicz]

Testing now. Just in case anyone else tries, if you run the biome extractor on an active multiplayer world it crashes the server. Hopefully they can fix this.

[agrif]

@eminence :

The sides of the grass can't really be tinted, right now, because Overviewer is designed to be used with any Minecraft texture pack. It kinda looks bad in-game right now anyway, so I bet that's somewhere on Notch's todo list.

If I could make a suggestion: I think the code for loading the biome data should be in the ChunkRenderer class. That way, you can avoid having the global variables and function arguments to prepareBiomeData(), and just use the self.* variables. Also, if Notch ever adds biome data to the chunk files themselves, it'd be an easy code fix.

I have an idea for the grass tinting that works similar to how I did lighting, and it might be faster. If I can find time to test it, I'll post results.

@alex : the biome extractor crashes the server? That's... wow. That's really bad. Hmmm...

[eminence]

@agrif,

Well, we can still tint the sides of the grass, it'd just be annoying (for example, mask out the dirt part, convert to grayscale, apply tint). I do agree, though, that we ought to wait to see how Notch addresses this issue.

About your suggestion: If I'm not mistaken, a ChunkRenderer is instantiated for every chunk to be rendered. If the biome-data loading stuff went into the ChunkRenderer, then it wouldn't be able to cache the biome data (each .biome file contains the biome data for 64 chunks. It would be good to save on opens and reads by not re-reading the same data 64 times). What do you think?

[agrif]

I was not aware that the biome files contain multiple chunks of info. I guess that makes sense, looking back.

It looks like overviewer renders chunks along one axis first, then the other. Row-by-row, I guess. So it will only read at most 8 chunks of info from each file before reading another. Perhaps a nice way to do it is to only read in the info you need, and seek across the rest, but maybe that's too complicated. shrug I don't know, just a suggestion.

Maybe even just reading the file for each chunk render would be OK? Consider that the chunk renderer writes out a ~40kb .png file, which will take much longer to write than reading the ~32kb biome file, without even considering the draw functions and png compression.

Again, I dunno. Just throwing ideas out there... it'd probably be best to keep the code where it is for now, as long as it works.

[alexjurkiewicz]

OK, it looks like your code works here too :) http://qt3minecraft.project357.com/map.htm My version of your code is in my master branch.

[eminence]

great. thanks for testing (and cleaning up the gearman cruft). probably the only thing left to do would be to add a command line option to enable biome rendering (it should probably be disabled by default). once that's done, can anyone think of a reason this code shouldn't be merged into brownan's master?

[alexjurkiewicz]

My opinion: biome rendering is not a user option. Biome data is used if available, and if not a warning is emitted and the program just tints everything a default shade. PS, I think we should include the biome code in the Overviewer repository, even if we don't use it directly. I've added it to my master branch. PPS, I also think lighting should become default.

[agrif]

For when biome rendering is off, you should use slightly differing colors for the leaves and grass. Before, I think they were the same and it was getting a little hard to tell them apart. Maybe you could get these colors from the biome pngs, at some fixed index?

You can probably load the biome image files out of the minecraft jar by using _load_image("misc/grasscolor.png") (and also for foliagecolor.png), which uses _find_file(), which should look automatically everywhere it looks for texture images. Haven't tried it, though.

Oh, and edit: I made lighting an optional thing so that people pulling the newest Overviewer version weren't surprised when their incremental map had dark spots, and so their cache wasn't ruined. I'm not sure how important that is now, though, with everyone and their mom using lighting.

[emirin]

@Eminence

I've been trying to debug this for a day now, I'm running into an issue when running your biome fix on line 573 of textures.py and I'm recieving a "ValueError: buffer size must be a multiple of element size". Any help on this?

[eminence]

emirin, what commit are you using?

[emirin]

I really haven't any idea, I'm not a python guy. I've been trying to run the branch you posted like you run the main branch of Overviewer. Point me in the direction you need me to go to figure this out, I've programed in a lot of languages, just haven't done python yet.

[eminence]

i mean what git commit are you using? I need to know (since line 574 in your version of textures.py might be different than mine). To find out, run:

git log -1

Also, can you paste the traceback?

Thanks.

[emirin]

Fyi I'm using python for windows and don't have any version control softwear installed, just used the zip download link for your respository on here on gitHub. Below is the postback from running gmap

c:\Minecraft>python C:\Minecraft\Overviewer-test\gmap.py -p 1 --cachedir="C:\ser
ver\www\Maptest\images" "C:\Minecraft\bin\Pingas" "C:\server\www\Maptest"
2010-11-17 11:35:01,562 [WARNING] alpha_over extension not found; using default
PIL paste()
2010-11-17 11:35:01,884 [INFO] Welcome to Minecraft Overviewer!
2010-11-17 11:35:01,884 [INFO] Scanning chunks
2010-11-17 11:35:08,832 [WARNING] alpha_over extension not found; using default
PIL paste()
Traceback (most recent call last):
  File "C:\Minecraft\Overviewer-test\chunk.py", line 122, in render_and_save
    return a.render_and_save(cave)
  File "C:\Minecraft\Overviewer-test\chunk.py", line 358, in render_and_save
    img = self.chunk_render(cave=cave)
  File "C:\Minecraft\Overviewer-test\chunk.py", line 500, in chunk_render
    self.chunkX, self.chunkY)
  File "C:\Minecraft\Overviewer-test\textures.py", line 573, in prepareBiomeData

    data = numpy.frombuffer(rawdata, dtype=numpy.dtype(">u2"))
ValueError: buffer size must be a multiple of element size
Traceback (most recent call last):
  File "C:\Minecraft\Overviewer-test\gmap.py", line 174, in <module>
    main()
  File "C:\Minecraft\Overviewer-test\gmap.py", line 121, in main
    w.go(options.procs)
  File "C:\Minecraft\Overviewer-test\world.py", line 214, in go
    self.chunkmap = self._render_chunks_async(chunks, procs)
  File "C:\Minecraft\Overviewer-test\world.py", line 278, in _render_chunks_asyn
c
    result = chunk.render_and_save(chunkfile, self.cachedir, self, cave=self.cav
es, queue=q)
  File "C:\Minecraft\Overviewer-test\chunk.py", line 122, in render_and_save
    return a.render_and_save(cave)
  File "C:\Minecraft\Overviewer-test\chunk.py", line 358, in render_and_save
    img = self.chunk_render(cave=cave)
  File "C:\Minecraft\Overviewer-test\chunk.py", line 500, in chunk_render
    self.chunkX, self.chunkY)
  File "C:\Minecraft\Overviewer-test\textures.py", line 573, in prepareBiomeData

    data = numpy.frombuffer(rawdata, dtype=numpy.dtype(">u2"))
ValueError: buffer size must be a multiple of element size

[alexjurkiewicz]

I've seen this too. It seems to only happen with --night for me. Here's the output of a night run: python gmap.py --imgformat=jpg -p2 --lighting --night --cachedir cachedir-night world/ map-night/ 2010-11-18 00:57:17,393 [INFO] Welcome to Minecraft Overviewer! 2010-11-18 00:57:17,450 [INFO] Scanning chunks Traceback (most recent call last): File "/mnt/qt3minecraft/Minecraft-Overviewer/chunk.py", line 122, in render_and_save return a.render_and_save(cave) File "/mnt/qt3minecraft/Minecraft-Overviewer/chunk.py", line 358, in render_and_save img = self.chunk_render(cave=cave) File "/mnt/qt3minecraft/Minecraft-Overviewer/chunk.py", line 615, in chunk_render composite.alpha_over(img, Image.blend(t[0], black_color, black_coeff), (imgx, imgy), t[1]) File "/usr/lib/python2.6/dist-packages/PIL/Image.py", line 2008, in blend return im1._new(core.blend(im1.im, im2.im, alpha)) ValueError: images do not match Traceback (most recent call last): File "/mnt/qt3minecraft/Minecraft-Overviewer/chunk.py", line 122, in render_and_save return a.render_and_save(cave) File "/mnt/qt3minecraft/Minecraft-Overviewer/chunk.py", line 358, in render_and_save img = self.chunk_render(cave=cave) File "/mnt/qt3minecraft/Minecraft-Overviewer/chunk.py", line 615, in chunk_render composite.alpha_over(img, Image.blend(t[0], black_color, black_coeff), (imgx, imgy), t[1]) File "/usr/lib/python2.6/dist-packages/PIL/Image.py", line 2008, in blend return im1._new(core.blend(im1.im, im2.im, alpha)) ValueError: images do not match Traceback (most recent call last): File "/mnt/qt3minecraft/Minecraft-Overviewer/chunk.py", line 122, in render_and_save return a.render_and_save(cave) File "/mnt/qt3minecraft/Minecraft-Overviewer/chunk.py", line 358, in render_and_save img = self.chunk_render(cave=cave) File "/mnt/qt3minecraft/Minecraft-Overviewer/chunk.py", line 615, in chunk_render composite.alpha_over(img, Image.blend(t[0], black_color, black_coeff), (imgx, imgy), t[1]) File "/usr/lib/python2.6/dist-packages/PIL/Image.py", line 2008, in blend return im1._new(core.blend(im1.im, im2.im, alpha)) ValueError: images do not match 2010-11-18 00:57:59,443 [INFO] 10/123805 chunks rendered 2010-11-18 00:57:59,956 [INFO] 20/123805 chunks rendered 2010-11-18 00:58:00,033 [INFO] 30/123805 chunks rendered 2010-11-18 00:58:00,047 [INFO] 40/123805 chunks rendered 2010-11-18 00:58:00,557 [INFO] 50/123805 chunks rendered 2010-11-18 00:58:00,564 [INFO] 60/123805 chunks rendered 2010-11-18 00:58:00,592 [INFO] 70/123805 chunks rendered 2010-11-18 00:58:00,599 [INFO] 80/123805 chunks rendered 2010-11-18 00:58:00,606 [INFO] 90/123805 chunks rendered 2010-11-18 00:58:00,673 [INFO] 100/123805 chunks rendered 2010-11-18 00:58:01,451 [INFO] 200/123805 chunks rendered 2010-11-18 00:58:01,461 [INFO] 300/123805 chunks rendered 2010-11-18 00:58:01,473 [INFO] 400/123805 chunks rendered 2010-11-18 00:58:02,011 [INFO] 500/123805 chunks rendered 2010-11-18 00:58:02,041 [INFO] 600/123805 chunks rendered Traceback (most recent call last): File "gmap.py", line 174, in main() File "gmap.py", line 121, in main w.go(options.procs) File "/mnt/qt3minecraft/Minecraft-Overviewer/world.py", line 214, in go self.chunkmap = self._render_chunks_async(chunks, procs) File "/mnt/qt3minecraft/Minecraft-Overviewer/world.py", line 317, in _render_chunks_async results[(col, row)] = result.get() File "/usr/lib/python2.6/multiprocessing/pool.py", line 422, in get raise self._value ValueError: images do not match

I'll try to get some more useful info for you.

[eminence]

Alex, you have confused me -- your traceback doesn't look anything like the one that emirin posted. Am I missing something?

[alexjurkiewicz]

Oh damn.. I skimmed the first error report on the assumption it was the same. You're right, this looks like another issue.

[eminence]

emirin, can you try re-running the Biome Extractor? If that doesn't fix the problem, can you please zip up the EXTRACTEDBIOMES folder and post it online?

[emirin]

http://bnminecraft.dyndns.org/Extractedbiomes.zip

Just ran the extractor 5 minutes ago after deleting all the old extracted biomes.

[emirin]

Any idea what I could be running into?