New Map file Format

[Ben0mega]

The newest Minecraft snapshot has a new map file format -details here http://www.mojang.com/2012/02/15/minecraft-snapshot-12w07a/ and here http://www.minecraftwiki.net/wiki/Upcoming_features

Assumedly it will be in the next release, so c10t will need to support it.

[uap-universe]

Yes - unfortunately the impact is high. I'm very interested in updating c10t not only for my own purposes, so be sure that c10t WILL be updated. But I can't promise an exact release date and I might need @udoprog's help. Of course any contribution from anyone else is really apreciated :)

[udoprog]

This is good news, since the changes are sane. Stuff like saving the biome in the map and increasing height, swapping axes will only improve both the quality of the rendering. Lighting will take a hit tho since it is completely rendered client-side.

[uap-universe]

It's really nice to hear from you :) Does that mean, that you will implement the changes on c10t just in time? If you are busy and don't have sooo much spare time for c10t I can offer some help if you want, just give out some tasks and I'll do it (testing and/or development, bringing a cup of virtual coffee and stuff like that ;)

[Pewmaymen]

I can test on Linux if you need it when you have more progress made. Just let me know.

[mathuin]

The solution to this issue will also resolve #208, so I will mark #208 closed.

[Tmkrth]

Hi all. How are things progressing?

[uap-universe]

Unfortunately it seems that neither @udoprog than me have the time right now. I'm really sad about this. The difficulty for me is, that I haven't analyzed udoprogs NBT parser yet. First I must invest some learning before I can change things. I hope I will have time for this in the next one or two weeks. I really want c10t back working guys, believe me :)

[arjunyg]

All of this is very very interesting. I am also the sort of person who really doesn't have the time to work on this size of a project, but I looks like either UniversE or udoprog will Eventually get it done. I kinda depend on c10t so yeah...

Also lighting is simple: flag all light sources when parsing somehow + flag all blocks in a 15x15 area with some sort of near-light flag if this program is implemented object-orientedly this shouldn't be a challenge

then when finally outputting the png, check those flags and add 15 levels to any source blocks + any blocks exposed to daylight (if daylight is enabled) or 4 to any block exposed to moonlight (for night renders) next add 14 to the neighboring blocks of the sources then 13 to the neighbors of the neighbors and so on!

this might be somewhat difficult, but it can't be that hard? at least the maps are still based in NBT xD

-arjunyg

[uap-universe]

To be honest, rewriting or writing a completly new map generator is less difficult for me than modifying c10t. It would be much easier if there was something like "MapParser.cpp" :D but @udoprog has just begun refactoring things and hasn't finished his work yet, so the code is still quite.... err... challanging :D

It seems that the code that must be modified is spread over the files in the src/mc/ directory. I still don't know where to start, but I think I (slowly) get an idea of what has been done there.

[uap-universe]

Sorry guys - i've given up.... I started writing handler for the new IntArray Tag but scince block_rotation can only handle byte arrays I'm really stuck. I cannot change it to IntArrays because other data (that do not belong to the height map) still remain byte arrays. I tried to split it up but .... yeah... i'm stuck :(

For me it seems to be the right approach to write a completly own map generation tool in java from scratch. I really dislike the idea of leaving c10t down but currently it looks like if I had to invest O(n!) work to get it back running. Sorry guys, really ... :(

EDIT: For those who might wanna continue the work. This is what I planned to do:

1. Adding IntArray to src/nbt/* files (a new type in types and parser etc. in nbt.hpp / cpp and so on)
2. Modifying block_rotation in some way that in can handle both ByteArray and IntArray and changing the XYZ order
3. Modifying src/mc/levels.cpp and anything belonging to it, so it has a register_int_array method and anything else it could need for anvil
4. Putting heightmap to the int_array handler and leaving anything else in the byte_array handler (I hope I have understood the new map format right - the documentation on minecraft wiki is really short)
5. adding some code that reads the enclosing Section List and calling the "old" parser for each y value. This means that the old method is encapsulated into another method which calls the old (but modified) parser up to 16 times with the Y value that can be read from file (multiplied with 16).

I don't know if this would work, but this is what I wanted to try out.

[udoprog]

a5eca2e60fa2dd1ad09d6e1126219675caf574b8 adds support for IntArray

To test, run

make region-inspect

and then

./region-inspect <mca-file>

on any .mca file, it should output something describing the region without problems.

[Ben0mega]

(I am a programmer but I am not sure where to start to help - mcedit, a 3d map viewer uses pymclevel (https://github.com/mcedit/pymclevel) which can parse the new format and could be used as a reference to make sure everything is going in the right direction)

[udoprog]

Thanks!

Parsing is however already solved, rendering is a whole other issue since they decided to order level data in a new manner.

[udoprog]

Allright, rendering kind of working again, anyone who wants to try it out should just compile the master branch.

Some things to note;

• Shading needs some serious overhaul, it changed a lot from the last release.
• Hell mode and cave mode have not been implemented since the rendering phase has been overhauled.
• Caching has been improved to compensate for terribly slow rendering. Use it!

[arjunyg]

At this point I assume anyone working on this project is familiar with the new map format, but for any new people or whomever here is an overview:

Lighting is not stored in the save files

Biomes are now in the files, this will allow prettier renders of grass, water and leaves.

The ordering of the block data has changed

Also some source code is available for download here that converts the mcregion map format to the anvil format.

http://www.mojang.com/2012/02/new-minecraft-map-format-anvil/

-arjunyg

On Apr 3, 2012, at 9:17 AM, John-John Tedro reply@reply.github.com wrote:

Allright, rendering kind of working again, anyone who wants to try it out should just compile the master branch.

Some things to note;

• Shading needs some serious overhaul, it changed a lot from the last release.
• Hell mode and cave mode have not been implemented since the rendering phase has been overhauled.
• Caching has been improved to compensate for terribly slow rendering. Use it!

---

Reply to this email directly or view it on GitHub: https://github.com/udoprog/c10t/issues/249#issuecomment-4907558

[udoprog]

Light has not been removed, just moved into the sections. See the "SkyLight" and "BlockLight" tags.

[arjunyg]

Really? I got the impression that it was removed from the update news that says light is calculated client-side.

I guess it's been separated from the regular chunk data, so the server doesn't hunt for it, it just sends the blocks. Actually Jeb probably left the lighting in the save file to help out people that make these third party tools. It does seem that Jeb has been thinking of modders a lot lately, such as the extra bits for block IDs etc.

Hopefully c10t will be updated/fixed soon :)

-arjunyg

On Apr 3, 2012, at 12:57 PM, John-John Tedro reply@reply.github.com wrote:

Light has not been removed, just moved into the sections. See the "SkyLight" and "BlockLight" tags.

---

Reply to this email directly or view it on GitHub: https://github.com/udoprog/c10t/issues/249#issuecomment-4924751

[udoprog]

snapshots available for testing; http://toolchain.eu/jenkins/job/c10t-snapshot/

[Pewmaymen]

I have to build for source to get things working due to the age of my operating system. How do I do that through toolchain or can you provide it there?

-----Original Message----- From: John-John Tedro [mailto:reply@reply.github.com] Sent: Wednesday, April 04, 2012 2:25 PM To: Pewmaymen Subject: Re: [c10t] New Map file Format (#249)

snapshots available for testing; http://toolchain.eu/jenkins/job/c10t-snapshot/

---

Reply to this email directly or view it on GitHub: https://github.com/udoprog/c10t/issues/249#issuecomment-4959793

[Pewmaymen]

Actually, I will see if I can update from github and build. Let me know if that won't work for some reason.

-----Original Message----- From: John-John Tedro [mailto:reply@reply.github.com] Sent: Wednesday, April 04, 2012 2:25 PM To: Pewmaymen Subject: Re: [c10t] New Map file Format (#249)

snapshots available for testing; http://toolchain.eu/jenkins/job/c10t-snapshot/

---

Reply to this email directly or view it on GitHub: https://github.com/udoprog/c10t/issues/249#issuecomment-4959793

[Pewmaymen]

I grabbed the source from github and compiled it without apparent issues. Running it does seem to see the appropriately larger map compared to the non-anvil code. However, it failed at the end of the render and during the image generation with the following output: ............................. 91384 parts 100% image limits: 0x15 to 18446744073709551615x18446744073709551614 will be the cropped image (18446744073709551615x18446744073709551599) --- SAVING IMAGE --- path: /downloads/shawn/minecraft_maps/isometric_hourly_1.0_survival.png libpng error: image size exceeds user limits in IHDR /downloads/shawn/minecraft_maps/isometric_hourly_1.0_survival.png: unknown libpng error /tickets/shawn/minecraft/c10t/c10t: Log written to /tickets/shawn/src/c10t/c10t/build/c10t.log

Those limits are WAY too big, so there are still some bounding issues that need resolving there.

-----Original Message----- From: John-John Tedro [mailto:reply@reply.github.com] Sent: Wednesday, April 04, 2012 2:25 PM To: Pewmaymen Subject: Re: [c10t] New Map file Format (#249)

snapshots available for testing; http://toolchain.eu/jenkins/job/c10t-snapshot/

---

Reply to this email directly or view it on GitHub: https://github.com/udoprog/c10t/issues/249#issuecomment-4959793

[Ben0mega]

I am having rendering artifacts with my build - see image at https://docs.google.com/open?id=0B0n2OEIlU0EZQXN6NEQ1UnRUa0dIaWp6bmFCMGtDQQ (specifically I have speckling that can easily be seen on the deserts and oceans, and holes in the image in random locations). Please ask if you need other data about the map/stdout

[uap-universe]

Yes, there are serious issues with the the rendering routines. You can see very ugly artifacts where torches should be. You can see that more cleary on an isometric rendering (e.g. on my server: http://minecraft.uap-core.de/map/ - click on the 2D image to get the isometric view). Furthermore rendering is very slow and I think it costs more memory than before. But I cannot prove that because my map has grown a bit. But I think 3,5 Gigabytes for a radius of 200 (without every chunk beeing generated!) is way to much, anyway.

I think @udoprog will get things working again (somehow :D)- but I'm definately out... I don't know whether the code is too cumbersome or too sophisticated for me (in dubio pro reo I think it's too sophisticated ;), but I don't get any idea behind this code right now. He will do a great job and c10t will be as wonderful as it was before the map format update - just be patient.

I'll concentrate on the development of MineMap - my map tool. It won't have as many feature as c10t has (for now), because I develop anything from scratch. The first goal is to get a very performant top-down rendering working (less than 3 seconds per region file on a night rendering - I am at 5 seconds right now) - then the source code goes public and anyone can feel free to copy the code or the ideas behind the code.

For the isometric view later I plan some kind of raytracing so that we can render tiles based on camera rays and don't need to render the whole image on the one hand, and to render just the blocks that need to be rendered instead of rendering any block from bedrock to air on the other hand. But this is something I think about in future.

Sorry guys, that all information to MineMap are in German right now on my weblog at www.uap-core.de, but things will change when I get something working that's worth showing you.

[Ben0mega]

Last push is a vast improvement. The only bugs I experience now are white and brown/orange (probably sand and dirt?) spots appearing in the ocean (the blueing of the water is not applied?).

[Psithief]

Having a look at the isometric rendering corruption, I have to ask whether the change in axes has been taken into account yet? It looks like all the chunks are being rendered in the wrong order.

[Ben0mega]

I am getting 'holes' in the snowy biomes - pure alpha.

(accidently closed - ignore)

[arjunyg]

is anyone still working on this? maybe someone who can will get a notification from this comment if they forgot XD I hope you guys can get this straightened out soon :/

[chum]

c10t was working with the new (Jungles biome) file format for a while but then, with the 1.3.1 update, we started getting assert errors. Should that be under this issue, or a new bug report? Is anyone working on it? (NOTE: running on Mac, though I don't know if that matters.)

[Ben0mega]

(I am not experiencing this new error) did you compile from trunk? so it is probably the addition of the cocoa beans if the assert is failing on only the new jungle biomes. trunk has some safety checks that just discard blocks that c10t doesnt understand. This is a kludge in the case of cocoa beans, but it will solve the immediate problem (but should have a true fix at some point).