Program does not open development beta Minecraft 18x10b

erich666 commented 6 years ago

The latest Minecraft beta, 18x10b, makes world files that cannot be opened by Minutor, which crashes. I suspect 18x10a, https://minecraft.gamepedia.com/18w10a, has the same problem, and my guess is that restructuring for the coral plants is causing problems. Given that Minecraft "used up" all the basic block IDs 0-255 (except for 253 and 254), https://minecraft.gamepedia.com/Java_Edition_data_values#Block_IDs, I suspect they're now using some new extension method that breaks readers - that'll be fun for everyone.

I hope to look into this problem in a bit (Mineways is built off an old version of minutor), but thought I'd give a heads-up and see if anyone's figured it out already.

mrkite commented 6 years ago

I haven't been keeping up with Minecraft, so this is definitely news to me. I'll have to find some time to install Minecraft and see where things go haywire.

EtlamGit commented 6 years ago

As far as I know, the next major release from Minecraft may bring a completely new storage format, but development is not finished now. I remember reading something like: worlds are converted on first load, old structures will not be converted properly, no way back, ...

erich666 commented 6 years ago

I finally had some time to look at this today (after recovering from updating to VS 2017). Yes, they're having fun with the file format. For starters, Biomes went from 256 bytes to 256 integers, and it looks like the fourth (high) byte of the integer is the biome value.

New format is on the left, as seen in NBT Explorer:

Sections are now pretty interesting, with each 256 slice having a palette associated with it. BlockStates are now 256 long integers. These integers are actually just a bunch of half-bytes if the palette of blocks is <= 16 palette entries (i.e. there are only 16 block types or less in the 16x16x16 Y-slice). I haven't decoded it beyond this - looks like a big puzzle. Palette entries now also hold the "data" orientation etc. information. Basically, a lot of what you know has changed. Since palettes are expandable to any number of entries, this lets them break through the 256 block limit - no blocks now have ID values per se, just names and Properties.

I decided to see what would happen if I added enough blocks to reach 29 palette entries, so that 4 bits would not be enough for each entry. The BlockStates expands to 320 long ints, i.e., 5 bits per entry. So palette size determines BlockStates's size and decode. Looks pretty straightforward, though tedious to decode by bytes, a "bit grabber" (e.g., "I want 5 bits") for the BlockState is probably better, but might be slow... Here's an example:

Anyway, I see that a lot of blocks are missing from the beta at this point (e.g., birch blocks are just birch bark, no interior), so I suspect the final release of 1.13 is far far away. I might look again in a long while, once it's all settled down - no reason to write a lot of code to support something they may revise.

nnooney commented 6 years ago

I've updated Minutor to parse the new format described above. Thanks @erich666 for providing the format that allowed me to understand the new format. The pull request I referenced is very rough around the edges, but I am able to open worlds that I generated in the 1.13 snapshots.

More work needs to be done to update the Definition Manager with the new block format, and there's some room for code cleanup. However, it should be enough to load worlds in Minecraft 1.13+ (I certainly broke backwards compatibility).

erich666 commented 6 years ago

And just in case, Minecraft 1.13 is now officially out, as of yesterday https://minecraft.gamepedia.com/1.13

EtlamGit commented 6 years ago

I found some time to look into the changed file format and to test the 1.13 branch from @nnooney .

Mojang's changes look straight forward, but not all new tags are clear at the moment. I have not reviewed the new code jet. Maps can be loaded and look more or less ok. It seems that some transparency is not working, e.g. flowers look different. Where did you find the values for watermodifier for the different ocean types?

But definitely some functionality is broken:

hostile mob spawn detection is not working at all (as expected)
all Structure overlays are missing (except some villages) (expected as now stored differently)
Entity overlays have to be cleaned up, but work mostly
1.13 branch Minutor is crashing with old maps
old Minutor crashes with 1.13 maps
automatic definition file update will break old Minutor releases

I would appreciate if Minutor could handle old and new map file format in one single version. For most users this might not be necessary, as they just migrate to 1.13 and never back. But I normally use different (old) versions for a longer time. In the next days I try to think about ways to get it compatible (together with @nnooney 's changes). Here my current brainstorm ideas for commenting:

stored tags in the files have version information, so we can distinguish them
basis for data structures in new Minutor versions should be based on the current format (so probably similar to the changes from @nnooney )
a translator/upgrade module is needed to convert old map data (numerical IDs) to the new format (string names), but only that part we need for rendering
this translator can be fixed code (no .json file), as numerical IDs will never change again
chunk reading together with block identification should be encapsulated more from map rendering. At the moment the rendering has knowledge about storage format. If we abstract that, the ChunkLoader could load arbitrary version map data and MapView can render based on an abstract interface.
to stay backwards compatible, we may have to rename all/some definition (.json) files. As otherwise older Minutor versions automatically upgrade to incompatible definitions.

erich666 commented 6 years ago

Since Tuesday I've been actively working on 1.13 translation for Mineways, so feel free to ask any questions (not that I necessarily have answers, but I know more than I did a few days ago). My code's a hack, but works to translate 1.13 content into 1.12.2 block and data values. For new blocks I'll be crowbarring these in by using high-order bits in the data values for new (made up by me) block IDs, since I use a full byte to store data values, not just 4 bits (lucky I was lazy that way). Anyway, my code is based on ancient minutor code, but it might be of help. I hope to check it in by next Wednesday, and am happy to share it with anyone before then.

EtlamGit commented 6 years ago

Hi @erich666,

the approach you describe sounds to me to translate new 1.13 string names to numerical (old) ID values. Obviously this needs less storage space. But I fear that this will (shorter or later) hit value range boundaries when new blocks are introduced. And maintaining a very long ID list without any official reference might result in much work and problems.

To stay (very long) future prove I would go the other way around: Translating old ID+data into string names. This will consume more memory and maybe reduce performance, but we will be able to follow any number of new blocks. And I expect many many new blocks in the next releases. The main advantage would be that this translation will be done only once. In the future there would be no need for further translation. Instead of the complete string we might use a hash of that string (which is more or less the same than an ID). Then we can use a hash-map to store all possible blocks. Access key is the hashed string and no longer the numerical ID. If we use the same string hashing function in Minutor and Mineways, we would get the same hash-ID.

madmaxoft commented 6 years ago

How about using something in between? The in-memory representation could more or less mimic the 1.13 format - each chunk's section would have an array of ints and a Palette object that would translate those numbers into instances of block type classes. Each block type class instance would thus represent all blocks that have the same string ID and the same properties (and an assigned color / transparency / ... for rendering). We'll probably be implementing something similar in Cuberite as well.

mrkite commented 6 years ago

I agree.. plus translating old maps into new maps will prevent us from having to update the translator for every future update... it's a one-and-done thing.

madmaxoft commented 6 years ago

So this is the pseudocode for the renderer reading a color / transparency value:

const auto & section = chunk->section[y / 16];
auto blockType = section->palette->resolve(section->blocks[blockIndex(x, z)]);
// Render using blockType.color and blockType.transparency etc.

EtlamGit commented 6 years ago

Problem: backwards compatibility with automatic definition file updates

Description: Minutor regularly checks for changed definition files here on GitHub. If we change the definition files, older Minutor versions will get them after some days. In case we introduce incompatible changes (as in the current m1.13 branch) this will break Minutor at the client PCs of the users. This will happen without them knowing why.

Solution:

We will NOT change the syntax in the file vanilla_ids.json
Instead changes from @nnooney will get into a new file, e.g. vanilla_blocks.json
The old file vanilla_ids.json can be used by the Converter. Up to now it contains IDs and data field to identify all old blocks. If we add a new tag ("flatname": "minecraft:air") we can define all translation in that file.

EtlamGit commented 6 years ago

@mrkite , could you explain the initial idea behind the Definition Packs?

During my current tests to change the code in a backwards compatible way, I have trouble to maintain handling of several enabled/disabled Packs. In the current m1.13 branch this feature is already broken, as the packs are not searched for blocks.

For my understanding, in the past there could be ambiguities for block IDs coming from plugins and feature packs. Therefore we needed complicated lists, and priority search to find the blocks. As Mojang is now using long names and namespaces (minecraft:air), I understood, that no collisions are possible anymore.

mrkite commented 6 years ago

@EtlamGit yeah the initial idea was that plugins might be incompatible with each other and the user would know which plugins he was using on which maps and could turn them on and off as needed.

Plus, mod makers could host their own definition files so we wouldn't have to do all the work ourselves.

It never really panned out. If I had to do it over, I'd use protobuf to load definitions and stick it all in a single file.

EtlamGit commented 6 years ago

@erich666, as you asked to ask questions...:

Do we render blocks differently if they have specific Properties? E.g. I found it for restone_ore -> lit:false/true or water and lava with different level (distance). Probably a lit furnace or repeater will also be stored in that way. Do you think we need to use this kind of Properties stored for some blocks in the palette for rendering in a different color? Example: https://minecraft.gamepedia.com/Mushroom_Block#Block_state

This is not possible at the moment, as only the block name is used to find the BlockIdentifier entry. We would also need to find a JSON description to define Properties resulting in different colors.

NBT Heighmaps Did you find any description/decoding for the Heighmaps that are stored for each Chunk? We could speedup rendering with knowing the highest non-air block per XZ position.

erich666 commented 6 years ago

Do we render blocks differently if they have specific Properties?

Yes, you'll perhaps want to do this, if you wish. While objects like carpet now have individual names, vs. data values for each color, there are some properties you may wish to use to modify objects' appearance: mushroom blocks could have a different top color (e.g., red vs. the "internal" color), maybe use fluid level to (slightly) change water color, use moisture for farmland, use age for wheat color. But, other than red and brown mushroom blocks there's not a large difference that I've noticed (yet - I'm still slogging through the blocks).

Did you find any description/decoding for the Heightmaps that are stored for each Chunk?

I don't use these in Mineways so I haven't looked.

erich666 commented 6 years ago

FWIW, I'm done (for now) with the 1.13 version of Mineways. My code's a hacky mess, for the most part you should avert your eyes. But, the nbt.cpp comments and snippets might be of use, as might the new biomes list.

If anyone figures out the rainfall for these biomes, and temperature values for the ocean biomes, please do let me know. I poked around a fair bit, I couldn't find them (yet).

erich666 commented 6 years ago

Instead of the complete string we might use a hash of that string (which is more or less the same than an ID). Then we can use a hash-map to store all possible blocks. Access key is the hashed string and no longer the numerical ID.

Yes, I did this for Mineways as I felt it didn't even need any testing to know it would be faster than testing against the 500+ item strings. My hash system is pretty trivial, see this code, and the 500+ names are listed just before that. I didn't check over the hashes generated to see if they were well-distributed - I should do that at some point, just to see if there's not some massive clump in a few lists. I did check the special case of "air" before doing any hash computation and search, figuring that a high percentage of blocks are likely to be air and deserve a quick out. Again, didn't test that assumption, yet.

If we use the same string hashing function in Minutor and Mineways, we would get the same hash-ID.

No need. Mineways has diverged (or, I should say, been left in the dust - I wish Mineways had multiprocess loading) by Minutor at this point, so use any hash function you like.

EtlamGit commented 6 years ago

I browsed your code already, and yes it is different...

By the way, vanilla Minecraft 1.13 has 8581 different Blocks. And they simply number them consecutive. Which means the internal number will change with every version. At the moment I try to find out which of these variants we can render the same way and for which we have to invent a way to distinguish them.

erich666 commented 6 years ago

By the way, vanilla Minecraft 1.13 has 8581 different Blocks.

The 594 block names I have listed here appear to be the entire set of unique names in 1.13. There are of course tons of variants - 1 turtle egg vs. 2 vs. 3 vs. 4, which way a stair piece is oriented, on and on - but I suspect that for mapping purposes these names are just about all you need. Your call, of course.

EtlamGit commented 6 years ago

e.g. Sandstone Stairs have a different bottom texture. To be correct we would even have to detect if they are facing downwards. And it is more obvious for the mushroom stem or the direction of the wood Logs that they look different from top. But you are right most of the variants are irrelevant for rendering. And I think we can start with only 593 Blocks itself to get a very good map. In your list the block "bed" is not present in 1.13 anymore. Therefore the official list would be 593 Blocks long. But I will also include backwards compatible descriptions to load old worlds.

EtlamGit commented 6 years ago

@thewierdnut did you notice that with 1.13 structures are not stored in *.dat files anymore, but inside the region files together with the Chunk data? Do you have any plans to update the Structure loading code? Otherwise I would have to start with that task...

cloudlion-ca commented 6 years ago

I've tried to use Minutor to load worlds in Microsoft Windows 10 and OSX using the available latest download files. I have tried to use that release with Minecraft version 1.13.1 .... I assume that's the latest version of Minecraft.

On Microsoft Windows 10 on three different computers the application loads and then just shuts down when I attempt to load a world. On OSX Minutor just loads up the checker-board screen and I can't see anything. I move the pointer around and it indicates unknown biomes.

I can't imagine that I'm the only person experiencing this adventure.

Please note: I've gone through the process of cleaning out the Window's registry of any reference to seancode. Downloading the available package and re-installing it with Admin set for Windows. That has no effect what so ever.

I really miss Minutor when it's working.

Thanks for all your efforts to the development crew.

Cheers!

mrkite commented 6 years ago

The pre-built binaries don't work with 1.13. You instead have to build Minutor from source (in particular, the m13 branch). Eventually I'll release new pre-built binaries, I just haven't had time.

EtlamGit commented 6 years ago

I just updated the Wiki with images rendered with latest color and Block definitions.

AdriaanRenting commented 6 years ago

I cloned m1.13 and built it on my MacOs 1.13.6 It runs for a bit but once I start scrolling the map it crashes quite quickly. Built with clang (Apple LLVM version 9.1.0 (clang-902.0.39.2) and Qt5.5

In debug mode it gets to memcpy(this->biomes, biomes->toIntArray(), sizeof(int)*biomes->length()); line 62 in chunk.cpp before going into some low level assembly in platformMemmove$VARIANT$Nehalem

I think with an "Unhandled toIntArray" error or message.

Is it possible that it is encountering chunks that have not been converted to the new format yet? Does minecraft do a conversion on the fly, or convert everything when you run the new version for the first time?

I'm not terribly familiar with Qt Creator or its debugger, I just happen to have it on my machine for one piece of software I occasionally need to support.

AdriaanRenting commented 6 years ago

I seem to have narrowed down my problem to 4 region files in the Overworld (and probably one in the Nether).

When I remove those 4 region files from the Overworld, minutor works as expected on the rest of the world. I could have some other kind of corruption going on, but it would then affect at least 5 region files without seemingly causing problems in the game. Any suggestions on how I could go about debugging if this is a minutor m1.13 issue or something else?

EtlamGit commented 6 years ago

Is it possible that it is encountering chunks that have not been converted to the new format yet? Does minecraft do a conversion on the fly, or convert everything when you run the new version for the first time?

Minecraft 1.13 converts Chunks during loading them the first time after "upgrade". So there is no complete world upgrade process. Chunks get updated when you travel "near" them. Your world might contain Chunks with several different versions, depending when you last visited them. Therefore we now have different parsers for different versions in Minutor.

When I remove those 4 region files from the Overworld, minutor works as expected on the rest of the world.

When I try to load a world where Minecraft is writing during that very moment, I also get crashes. This is normally due to corrupt data in some region files. By traveling through these Chunks/Regions and than quit Minecraft the regular way before loading the world in Minutor it should work again.

AdriaanRenting commented 6 years ago

Is it possible that it is encountering chunks that have not been converted to the new format yet? Does minecraft do a conversion on the fly, or convert everything when you run the new version for the first time?

Minecraft 1.13 converts Chunks during loading them the first time after "upgrade". So there is no complete world upgrade process. Chunks get updated when you travel "near" them. Your world might contain Chunks with several different versions, depending when you last visited them. Therefore we now have different parsers for different versions in Minutor.

The region files I have trouble with, likely all contain at least some chunks that have been touched by 1.13 then. Everything pre1.13 seems to load. But quite a few post1.13 regions, including where the main buildings are on our server, seem to be fine as well.

Is there any way to easily narrow my issues down to a certain chunk or block? To me it seems that it is trying to read a chunk, but getting a wrong size for the memcpy or something like that. But I don't know enough about the chunk format to understand if my interpretation is correct.

When I remove those 4 region files from the Overworld, minutor works as expected on the rest of the world.

When I try to load a world where Minecraft is writing during that very moment, I also get crashes. This is normally due to corrupt data in some region files. By travelling through these Chunks/Regions and than quit Minecraft the regular way before loading the world in Minutor it should work again.

I only work on files that are copied from our server after I have stopped the minecraft server, so I assume that is not the issue. I understand that minutor cannot be reliable on files that are open and being written to by another program.

jamesmule commented 6 years ago

I'm having the same problem with a few of my worlds. It seems the biome array is supposed to be an int array in the new versions and a byte array in the old ones, but in the chunks causing the errors, the DataVersion tag that's checked for the case differentiation doesn't match the array type, i.e. it's a newer version, but the biome data is still saved in a byte array.

EDIT: Meaning the DataVersion tag is useless for checking what array type the biomes are saved in.

AdriaanRenting commented 6 years ago

I'm having the same problem with a few of my worlds. It seems the biome array is supposed to be an int array in the new versions and a byte array in the old ones, but in the chunks causing the errors, the DataVersion tag that's checked for the case differentiation doesn't match the array type, i.e. it's a newer version, but the biome data is still saved in a byte array.

EDIT: Meaning the DataVersion tag is useless for checking what array type the biomes are saved in.

That matches with what I'm seeing.

EtlamGit commented 6 years ago

Could you provide the non-working region file?

jamesmule commented 6 years ago

I don't have any more chunks with the inconsistency, so I manually deleted the int array and replaced it with a byte array at 0,0 in this one, but that shouldn't matter.

r.0.0.mca.zip

EtlamGit commented 6 years ago

A self modified region file is not the same than a real one... I would like to figure out what other data is around that int array in case of these failing Chunks.

@AdriaanRenting any region files from you?

And we should move discussion about this issue to #143.

AdriaanRenting commented 6 years ago

A self modified region file is not the same than a real one... I would like to figure out what other data is around that int array in case of these failing Chunks.

@AdriaanRenting any region files from you?

And we should move discussion about this issue to #143.

I added what I have to #143

erich666 commented 5 years ago

By the way, as a heads up, there is one way that beta-1.14 region files are different than 1.13. There are now up to 18 horizontal sections and these can have -1 or 16 (out of bounds normally) for the Y value. The -1 and 16 sections don't really have anything in them - blocks need to be in sections 0-15 - but these additional blocks were unexpected by Mineways' reader and crashed it. You get 18 sections if the region has something built up to the top of the sky.

EtlamGit commented 5 years ago

@erich666 thank you for that info. I will create a test world to trigger that...

mtantuan commented 4 years ago

So how does a non-developer try to even understand this. :(

mrkite / minutor

Program does not open development beta Minecraft 18x10b #125