Add support for many versions of Minecraft

[newtykip]

Create versions of the DataExtractor for various versions of Minecraft so that we can provide feature flags that provide all of the relevant blocks for a given version of Minecraft - also update the litematica schematic where relevant if necessary.

[RubixDev]

I think before going through the effort of writing and adapting the data extractor(s) for multiple MC versions, we should look into existing alternatives. The block state list is relatively easy to get, Mojang even provides a tool themselves, or otherwise possibly pixlyzer or preferably ArcticData (see the blocks and their properties) could be used.

The entity NBT structure is sadly not as easy to get and the current method is also far from flawless. The only two strategies I can really think of are:

1. decompiling and parsing the MC source code (the current approach)
2. scraping the info from the MC wiki

Neither is perfect. The main issue I see with the wiki route is it only being for the most recent MC version. For the source parsing we need info about all entity classes (currently part of the DataExtractor's job), but that at least seems to be partially available in pixlyzer but not in ArcticData (to be fair, this is dependent on the mappings used, yarn vs official).

Even apart from all that, I feel like the entire list stuff of the crate could do with a rewrite. (And docs are obviously needed, but that's besides the point)

These are just some thoughts, and before I commit to anything, it would be great to get some feedback :smile:

[newtykip]

The wiki is pretty much immediately out of the question, and if we are going to have to make a data extractor then shouldn't we just get the block state directly from it ourselves as well? That's dead simple compared to the entity data. For ease of use, we can download client jars and deobfuscation maps using the launcher's version manifest and grab the necessary dependencies for deobfuscation as follows (SpecialSource 1.10.0 should be good enough)

• SpecialSource 1.10.0
• Guava 20.0
• JOpt Simple 4.9
• ASM Commons 9.1
• ASM 9.1
• ASM Tree 9.1
• ASM Analysis 9.1

As for deobfuscation, why did you also try to use Fernflower? I never had any issues with CFR. If Fernflower had a use that has just flown past me, then setting that up will be a tad harder as we have to first clone the repository and then run gradlew build and grab the output from build/libs/fernflower.jar. CFR 0.152 is easy enough.

Maybe there is a smarter way that neither of us is noticing at first glance though because this does seem clunky. But not all problems have elegant solutions.

[newtykip]

A kind of janky but functional alternative I thought of is that you could always just use the latest Minecraft version's data and use ArticData's lists to choose which things to export. However, the reason this is janky is that there may be useless NBT fields or ones that have been renamed (idk if this has ever happened?)

Could be something we could work off of though?

[RubixDev]

Sorry for not responding earlier.

if we are going to have to make a data extractor then shouldn't we just get the block state directly from it ourselves as well? That's dead simple compared to the entity data.

true

For ease of use, we can download client jars and deobfuscation maps using the launcher's version manifest and grab the necessary dependencies for deobfuscation as follows [...]

I actually kinda like the current fabric mod approach, it already handles all the dependency stuff. I don't see why we should change that.

As for deobfuscation, why did you also try to use Fernflower? I never had any issues with CFR.

I always had one or two classes that the java-parser (used in the source-parser) couldn't parse as valid java code, so I used the FernFlower decompilation as a fallback which worked so far:

https://github.com/RubixDev/rustmatica/blob/b38529cac01e4c48651e3b2efd2dd01513d8c465/source-parser/src/main.ts#L22-L38

If Fernflower had a use that has just flown past me, then setting that up will be a tad harder as we have to first clone the repository and then run gradlew build and grab the output from build/libs/fernflower.jar. CFR 0.152 is easy enough.

Again, using a fabric mod setup handles it all for us.

Maybe there is a smarter way that neither of us is noticing at first glance though because this does seem clunky. But not all problems have elegant solutions.

Who knows :shrug:

---

Anyways, I've started work locally on rewriting the data extraction from scratch, intending to use the same strategies already used. I currently only have the block state list extraction, but that for all major Minecraft versions since 1.14.4 using three slightly different DataExtractor classes. To generate mod templates for the different versions, I depend on the same code as the Fabric CLI tools. That means I also use Deno instead of NodeJS. The code to run it all is now also written in Rust instead of Bash and will become an xtask.

The future cargo feature list for rustmatica (name, MC version used for extraction, DataExtractor number)

```json [ { "name": "1.14", "mc": "1.14.4", "extractor": 0 }, { "name": "1.15", "mc": "1.15.2", "extractor": 0 }, { "name": "1.16", "mc": "1.16.5", "extractor": 0 }, { "name": "1.17/1.18", "mc": "1.18.2", "extractor": 1 }, { "name": "1.19", "mc": "1.19.2", "extractor": 1 }, { "name": "1.19.3", "mc": "1.19.3", "extractor": 2 }, { "name": "1.19.4", "mc": "1.19.4", "extractor": 2 }, { "name": "1.20", "mc": "1.20.1", "extractor": 2 } ] ```

The DataExtractor number 2

```java package com.example; import com.google.gson.Gson; import com.google.gson.JsonArray; import com.google.gson.JsonObject; import java.io.FileWriter; import java.io.IOException; import java.util.*; import java.util.stream.Collectors; import net.fabricmc.api.ModInitializer; import net.fabricmc.fabric.api.event.lifecycle.v1.ServerLifecycleEvents; import net.minecraft.core.registries.BuiltInRegistries; import net.minecraft.util.StringRepresentable; import net.minecraft.world.flag.FeatureFlags; import net.minecraft.world.level.block.Block; import net.minecraft.world.level.block.state.properties.*; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class DataExtractor implements ModInitializer { public static final Logger LOGGER = LoggerFactory.getLogger("data-extractor"); @Override public void onInitialize() { Gson gson = new Gson(); LOGGER.info("Getting block info"); Map> enums = new HashMap<>(); JsonArray blocks = new JsonArray(); for (Block block : BuiltInRegistries.BLOCK) { JsonObject blockInfo = new JsonObject(); String blockId = BuiltInRegistries.BLOCK.getKey(block).toString(); blockInfo.addProperty("id", blockId); // check whether the block is experimental if (FeatureFlags.isExperimental(block.requiredFeatures())) { blockInfo.addProperty("experimental", true); } JsonArray properties = new JsonArray(); for (Property property : block.defaultBlockState().getProperties()) { JsonObject propertyInfo = new JsonObject(); propertyInfo.addProperty("name", property.getName()); if (property instanceof BooleanProperty) { propertyInfo.addProperty("type", "bool"); } else if (property instanceof IntegerProperty) { propertyInfo.addProperty("type", "int"); // the min and max fields are private, so we have to find them ourselves from all possible values Collection values = ((IntegerProperty) property).getPossibleValues(); if (values.isEmpty()) LOGGER.error("int property '" + property.getName() + "' of block '" + blockId + "' has no possible values"); propertyInfo.addProperty( "min", values.stream().mapToInt(v -> v).min().orElseThrow(NoSuchElementException::new)); propertyInfo.addProperty( "max", values.stream().mapToInt(v -> v).max().orElseThrow(NoSuchElementException::new)); } else if (property instanceof EnumProperty) { propertyInfo.addProperty("type", "enum"); // some enums have the same name but not the same values, we have to manually // assign custom names in those cases String enumName; if (property == BlockStateProperties.HORIZONTAL_AXIS) { enumName = "HorizontalAxis"; } else if (property == BlockStateProperties.HORIZONTAL_FACING) { enumName = "HorizontalDirection"; } else if (property == BlockStateProperties.VERTICAL_DIRECTION) { enumName = "VerticalDirection"; } else if (property == BlockStateProperties.FACING_HOPPER) { enumName = "HopperDirection"; } else if (property == BlockStateProperties.RAIL_SHAPE_STRAIGHT) { enumName = "StraightRailShape"; } else { enumName = property.getValueClass().getSimpleName(); } propertyInfo.addProperty("enum", enumName); List enumValues = property.getPossibleValues().stream() .map(value -> (value instanceof StringRepresentable) ? ((StringRepresentable) value).getSerializedName() : value.toString()) .collect(Collectors.toList()); if (enums.containsKey(enumName) && !enums.get(enumName).equals(enumValues)) { LOGGER.error("Ambiguous enum name: property '" + property.getName() + "' of '" + blockId + "' has name '" + enumName + "' with the values " + enumValues + ", but another enum with the same name has the values " + enums.get(enumName)); } else { enums.put(enumName, enumValues); } } else { LOGGER.error("unknown property type '" + property.getClass().getSimpleName() + "' with value type '" + property.getValueClass().getSimpleName() + "'"); } properties.add(propertyInfo); } blockInfo.add("properties", properties); blocks.add(blockInfo); } JsonArray enumList = new JsonArray(); enums.forEach((key, value) -> { JsonObject enumInfo = new JsonObject(); enumInfo.addProperty("name", key); enumInfo.add("values", gson.toJsonTree(value).getAsJsonArray()); enumList.add(enumInfo); }); LOGGER.info("Getting block property enums info"); JsonObject blocksJson = new JsonObject(); blocksJson.add("blocks", blocks); blocksJson.add("enums", enumList); LOGGER.info("Writing blocks.json"); try (FileWriter writer = new FileWriter("blocks.json")) { writer.write(gson.toJson(blocksJson)); } catch (IOException e) { throw new RuntimeException(e); } // entities must be spawned to inspect, which requires a world ServerLifecycleEvents.SERVER_STARTED.register(server -> { // TODO: entity and tile entity stuff LOGGER.info("Done!"); server.halt(false); }); } } ```

One additional complication I stumbled upon is the introduction of experimental features in 1.19.3. This includes blocks, for which this is easily detected and the info is already saved in the blocks.json, but it can also be used for anything else. For instance, since 1.20 Mob heads placed on note blocks play the respective mod sound, but this feature was introduces experimentally in 1.19.3, so the extraction output from 1.19.3 lists these sounds for the NoteBlockInstrument property, and this can not easily be detected from code. (Note that ArcticData also includes these in the 1.19.3 data). I think the right way to go about this is to have these things included in the appropriate rustmatica feature because they are in the game and schematics from the version could include them, but we should also document the fact that they were still experimental. Or what do you think?

I will probably push a new branch relatively soon with my current progress.

[newtykip]

I honestly have no idea why I thought we needed the client jars in that initial comment, we really don't. So I'm just going to ignore the fact I spoke about that and blame it on a lack of sleep 💀

Upon reflection, I think I was waffling about automating fetching the source code for the java parser and getting decompilers ready But I don't know why I was talking about the client jars, you need the yarn mappings anyway. I dunno what I was on.

Was FernFlower also not good enough on its own? Did that still leave a couple of faulty classes?

Anywho, I agree with your approach to experimental features, and the implementation details seem rugged. Good work so far (:

[RubixDev]

Was FernFlower also not good enough on its own? Did that still leave a couple of faulty classes?

Yes, either one on its own didn't suffice, but using both did.

Anywho, I agree with your approach to experimental features, and the implementation details seem rugged. Good work so far (:

Good!

[RubixDev]

Just to let you know, I've actually finally gotten back to working on this and have found a very promising way to get the entity NBT structure. It's now also a separate crate called mcdata as it isn't strictly just useful for rustmatica. I hope to get a new version of rustmatica with mcdata support released somewhat soon.

[RubixDev]

I believe I can close this now. rustmatica now uses mcdata and also finally has some documentation: https://docs.rs/rustmatica/ :tada: