search

piegamesde / BlockMap

An out-of-game map renderer and viewer for Minecraft 1.13–1.18 worlds [unmaintained]
MIT License
91 stars 22 forks source link

Support Minecraft 1.17 #63

Closed piegamesde closed 3 years ago

piegamesde commented 3 years ago

I'm actually a bit surprised nobody has asked for it yet. Anyways, the process is documented and only takes a few hours.

uncovery commented 3 years ago

I'm on this now but I am getting this:

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
java.lang.IllegalArgumentException: Did not find internal color map default
        at de.piegames.blockmap.color.BlockColorMap.loadInternal(BlockColorMap.java:348)
        at de.piegames.blockmap.color.BlockColorMap$InternalColorMap.lambda$getColorMap$0(BlockColorMap.java:54)
        at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180)
        at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        at de.piegames.blockmap.color.BlockColorMap$InternalColorMap.getColorMap(BlockColorMap.java:54)
        at de.piegames.blockmap.renderer.RenderSettings.loadDefaultColors(RenderSettings.java:46)
        at de.piegames.blockmap.renderer.RenderSettings.<init>(RenderSettings.java:29)
        at de.piegames.blockmap.standalone.CommandLineMain$CommandRender.call(CommandLineMain.java:196)
        at de.piegames.blockmap.standalone.CommandLineMain$CommandRender.call(CommandLineMain.java:113)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
        at picocli.CommandLine.access$1100(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
        at picocli.CommandLine.execute(CommandLine.java:2058)
        at de.piegames.blockmap.standalone.CommandLineMain.mainWithoutQuit(CommandLineMain.java:399)
        at de.piegames.blockmap.standalone.CommandLineMain.main(CommandLineMain.java:403)
Caused by: java.lang.NullPointerException
        at java.base/java.io.Reader.<init>(Reader.java:167)
        at java.base/java.io.InputStreamReader.<init>(InputStreamReader.java:72)
        at de.piegames.blockmap.color.BlockColorMap.loadInternal(BlockColorMap.java:346)
        ... 23 more
piegamesde commented 3 years ago

Very cool, thank you! You can change line 348 in BlockColorMap.java to the following to get a slightly better error message:

            throw new IllegalArgumentException("Did not find internal color map " + name + " (resource file 'block-colors-" + name + "-" + version.fileSuffix + ".json')", e);

Basically, it can't find the appropriate resources file for rendering. This is probably caused by one of two things: a) you need to regenerate b) you somehow got the name wrong in step two of the name process (again, regenerate afterwards).

uncovery commented 3 years ago

Ok I could fix it now. Likely the issue was that I did "clean and build" instead of "build' in my IDE (Netbeans). There are other errors now though:

tons and tons of this here:

21-07-15 18:07:13 WARN  ChunkRenderer_1_17 [176] - Failed to render chunk (31, 22) section 2. This is very likely because your chunk is corrupt. If possible, please verify it manually before sending a bug report.
java.lang.NullPointerException: null
21-07-15 18:07:13 WARN  ChunkRenderer_1_17 [219] - Failed to render chunk (31, 23)
java.util.NoSuchElementException: No value present
        at java.util.Optional.get(Optional.java:143) ~[?:?]
        at de.piegames.blockmap.renderer.ChunkRenderer_1_17.lambda$renderChunk$4(ChunkRenderer_1_17.java:63) ~[BlockMap-core-2.2.1.jar:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) ~[?:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276) ~[?:?]
        at java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
        at de.piegames.blockmap.renderer.ChunkRenderer_1_17.renderChunk(ChunkRenderer_1_17.java:60) [BlockMap-core-2.2.1.jar:?]
        at de.piegames.blockmap.renderer.RegionRenderer.renderRaw(RegionRenderer.java:143) [BlockMap-core-2.2.1.jar:?]
        at de.piegames.blockmap.renderer.RegionRenderer.render(RegionRenderer.java:60) [BlockMap-core-2.2.1.jar:?]
        at de.piegames.blockmap.world.RegionFolder$WorldRegionFolder.render(RegionFolder.java:180) [BlockMap-core-2.2.1.jar:?]
        at de.piegames.blockmap.world.RegionFolder$CachedRegionFolder.render(RegionFolder.java:468) [BlockMap-core-2.2.1.jar:?]
        at de.piegames.blockmap.gui.RenderedMap.run(RenderedMap.java:105) [BlockMap-gui-windows%2010.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
uncovery commented 3 years ago

This is what the GUI looks like image

piegamesde commented 3 years ago

They must have changed something on the world format, because I'm getting lots of funny exceptions. I'll have a look at this, this should be solvable independently of the 1.17 block colors support. We will need to coordinate a bit however: I created a minecraft-1-17 branch for this. Please make a pull request with your changes so far against that branch.

piegamesde commented 3 years ago

I pushed some commits on the branch, you should be good to go.

uncovery commented 3 years ago

ok this works now without errors. of course the new material colors are still missing. Also, the gradle task generateScreenshots does not work. I am doing this on windows and I get 

cd [xxx]\BlockMap; .\gradlew.bat --configure-on-demand -w -x check generateScreenshots
Configuration on demand is an incubating feature.
21-07-16 19:19:47 INFO  Generator [231] - Output path [xxx]\BlockMap\BlockMap-internal\.\build\generated-resources
21-07-16 19:19:47 DEBUG Generator [232] - Local resources path: file:/[xxx]/BlockMap/BlockMap-internal/build/classes/java/main/
21-07-16 19:19:47 INFO  Generator [195] - Generating test world
21-07-16 19:19:48 INFO  Server [46] - Server directory: [yyy]\AppData\Local\Temp\MinecraftLandGenerator3213361546723881050
21-07-16 19:19:48 DEBUG Server [47] - Symlinking server
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (public void de.piegames.blockmap.generate.Generator.generateTestWorld() throws java.io.IOException,java.lang.InterruptedException): java.nio.file.FileSystemException: [yyy]\AppData\Local\Temp\MinecraftLandGenerator3213361546723881050\server.jar: A required privilege is not held by the client
    at picocli.CommandLine.executeUserObject(CommandLine.java:1966)
    at picocli.CommandLine.access$1100(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2152)
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2530)
    at picocli.CommandLine.parseWithHandler(CommandLine.java:2465)
    at de.piegames.blockmap.generate.Generator.main(Generator.java:245)
Caused by: java.nio.file.FileSystemException: [yyy]\AppData\Local\Temp\MinecraftLandGenerator3213361546723881050\server.jar: A required privilege is not held by the client
    at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
    at java.base/sun.nio.fs.WindowsFileSystemProvider.createSymbolicLink(WindowsFileSystemProvider.java:595)
    at java.base/java.nio.file.Files.createSymbolicLink(Files.java:1063)
    at de.piegames.mlg.Server.<init>(Server.java:48)
    at de.piegames.blockmap.generate.Generator.generateTestWorld(Generator.java:202)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:567)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1952)
    ... 8 more

FAILURE: Build failed with an exception.

Do I need to run my IDE with admin privileges?

piegamesde commented 3 years ago

You don't really need to run generateScreenshots, that's not relevant for the actual process. But if you want to dig into this, it looks like MinecraftLandGenerator does not really work on Windows. (The software creates a temporary directory and symlinks things there for Minecraft, but apparently Windows is not happy about this somehow.

uncovery commented 3 years ago

Also just tried to run the CLI on linux and it's 

Error: Unable to initialize main class de.piegames.blockmap.standalone.CommandLineMain
Caused by: java.lang.NoClassDefFoundError: io/gsonfire/PostProcessor
piegamesde commented 3 years ago

Hm, works for me :/ Try if it also fails on master/latest release, if so, it'd be worthy of a separate issue. Otherwise, the breakage must have been caused by one of the recent commits and it shouldn't be hard to find the culprit.

uncovery commented 3 years ago

Actually that was me being stupid, I used the unshaded cli jar. Using the right one, it works for a lot of region fies but with a lot of errors of this sort:

command is

/etc/alternatives/jre_16/bin/java -jar /home/minecraft/server/blockmap/BlockMap.jar -v render /home/minecraft/server/worlds_save/empire --create-big-image --min-X=-2048 --max-X=2048 --min-Z=-2048 --max-Z=2048 -o /home/minecraft/server/maps/empire/png --create-tile-html

21-07-16 22:40:40 WARN  PostProcessing [146] - Could not load image (2 -4)
java.nio.file.NoSuchFileException: /home/minecraft/server/maps/empire/png/r.2.-4.png
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:375) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:426) ~[?:?]
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
        at java.nio.file.Files.newInputStream(Files.java:160) ~[?:?]
        at de.piegames.blockmap.world.RegionFolder$LocalRegionFolder.getInputStream(RegionFolder.java:362) ~[BlockMap-Beta.jar:?]
        at de.piegames.blockmap.world.RegionFolder$LocalRegionFolder.getInputStream(RegionFolder.java:354) ~[BlockMap-Beta.jar:?]
        at de.piegames.blockmap.world.RegionFolder$SavedRegionFolder.render(RegionFolder.java:319) ~[BlockMap-Beta.jar:?]
        at de.piegames.blockmap.world.RegionFolder$CachedRegionFolder.render(RegionFolder.java:465) ~[BlockMap-Beta.jar:?]
        at de.piegames.blockmap.standalone.PostProcessing.createBigImage(PostProcessing.java:144) [BlockMap-Beta.jar:?]
        at de.piegames.blockmap.standalone.CommandLineMain$CommandRender.call(CommandLineMain.java:269) [BlockMap-Beta.jar:?]
        at de.piegames.blockmap.standalone.CommandLineMain$CommandRender.call(CommandLineMain.java:113) [BlockMap-Beta.jar:?]
        at picocli.CommandLine.executeUserObject(CommandLine.java:1933) [BlockMap-Beta.jar:?]
        at picocli.CommandLine.access$1100(CommandLine.java:145) [BlockMap-Beta.jar:?]
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332) [BlockMap-Beta.jar:?]
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2326) [BlockMap-Beta.jar:?]
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2291) [BlockMap-Beta.jar:?]
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159) [BlockMap-Beta.jar:?]
        at picocli.CommandLine.execute(CommandLine.java:2058) [BlockMap-Beta.jar:?]
        at de.piegames.blockmap.standalone.CommandLineMain.mainWithoutQuit(CommandLineMain.java:399) [BlockMap-Beta.jar:?]
        at de.piegames.blockmap.standalone.CommandLineMain.main(CommandLineMain.java:403) [BlockMap-Beta.jar:?]
uncovery commented 3 years ago

Pls let me know if there is anything else I can do to help

piegamesde commented 3 years ago

I'd appreciate some help, yes. I've pushed some commits with various fixes and improvements to the branch (not all actually related to Minecraft 1.17 but I'll fix up the history later on).

The next step on the procedure is "Run all the tests and make them work", especially:

- If Minecraft added new blocks, the color map needs to be updated. The failing tests will tell which blocks are missing. Additional information can be retrieved from the default resource pack.
- If Minecraft added or changed biomes, manual checking and updating is required

The tests can be run with ./gradlew test (or individually via your IDE), and don't forget to regenerate the resources after every change.

uncovery commented 3 years ago

Ok so there is now obviously a list of blocks that was not found:

java.lang.AssertionError: Some blocks are missing expected:<[]> but was:<[minecraft:purple_candle_cake, minecraft:deepslate_tiles, minecraft:deepslate_tile_stairs, minecraft:yellow_candle_cake, minecraft:yellow_candle, minecraft:exposed_cut_copper_slab, minecraft:waxed_oxidized_cut_copper_stairs, minecraft:deepslate_copper_ore, minecraft:tuff, minecraft:waxed_oxidized_copper, minecraft:pink_candle, minecraft:light, minecraft:calcite, minecraft:magenta_candle_cake, minecraft:exposed_cut_copper, minecraft:candle_cake, minecraft:glow_lichen, minecraft:brown_candle, minecraft:flowering_azalea_leaves, minecraft:waxed_oxidized_cut_copper_slab, minecraft:black_candle_cake, minecraft:light_blue_candle_cake, minecraft:waxed_copper_block, minecraft:waxed_oxidized_cut_copper, minecraft:pointed_dripstone, minecraft:water_cauldron, minecraft:large_amethyst_bud, minecraft:white_candle_cake, minecraft:green_candle_cake, minecraft:weathered_cut_copper_stairs, minecraft:waxed_exposed_cut_copper_stairs, minecraft:potted_flowering_azalea_bush, minecraft:gray_candle, minecraft:small_dripleaf, minecraft:moss_block, minecraft:cave_vines, minecraft:cut_copper, minecraft:orange_candle_cake, minecraft:waxed_weathered_copper, minecraft:oxidized_cut_copper_slab, minecraft:powder_snow, minecraft:azalea, minecraft:lightning_rod, minecraft:cyan_candle, minecraft:polished_deepslate_wall, minecraft:waxed_exposed_cut_copper_slab, minecraft:waxed_weathered_cut_copper_slab, minecraft:black_candle, minecraft:infested_deepslate, minecraft:weathered_cut_copper_slab, minecraft:sculk_sensor, minecraft:purple_candle, minecraft:cracked_deepslate_tiles, minecraft:tinted_glass, minecraft:polished_deepslate, minecraft:deepslate_iron_ore, minecraft:polished_deepslate_stairs, minecraft:exposed_cut_copper_stairs, minecraft:waxed_cut_copper_stairs, minecraft:moss_carpet, minecraft:cobbled_deepslate_stairs, minecraft:blue_candle_cake, minecraft:chiseled_deepslate, minecraft:hanging_roots, minecraft:polished_deepslate_slab, minecraft:amethyst_block, minecraft:big_dripleaf, minecraft:waxed_weathered_cut_copper, minecraft:oxidized_cut_copper_stairs, minecraft:big_dripleaf_stem, minecraft:dirt_path, minecraft:deepslate_emerald_ore, minecraft:small_amethyst_bud, minecraft:light_blue_candle, minecraft:deepslate_brick_slab, minecraft:cave_vines_plant, minecraft:smooth_basalt, minecraft:brown_candle_cake, minecraft:gray_candle_cake, minecraft:lava_cauldron, minecraft:cut_copper_stairs, minecraft:red_candle_cake, minecraft:light_gray_candle, minecraft:orange_candle, minecraft:copper_block, minecraft:weathered_copper, minecraft:exposed_copper, minecraft:candle, minecraft:waxed_exposed_cut_copper, minecraft:cobbled_deepslate_wall, minecraft:flowering_azalea, minecraft:powder_snow_cauldron, minecraft:light_gray_candle_cake, minecraft:weathered_cut_copper, minecraft:cauldron, minecraft:pink_candle_cake, minecraft:deepslate_redstone_ore, minecraft:azalea_leaves, minecraft:lime_candle_cake, minecraft:cracked_deepslate_bricks, minecraft:deepslate_lapis_ore, minecraft:blue_candle, minecraft:raw_iron_block, minecraft:deepslate_brick_wall, minecraft:dripstone_block, minecraft:raw_gold_block, minecraft:cyan_candle_cake, minecraft:green_candle, minecraft:waxed_cut_copper, minecraft:white_candle, minecraft:spore_blossom, minecraft:budding_amethyst, minecraft:oxidized_copper, minecraft:deepslate_diamond_ore, minecraft:deepslate_brick_stairs, minecraft:deepslate_coal_ore, minecraft:red_candle, minecraft:amethyst_cluster, minecraft:waxed_weathered_cut_copper_stairs, minecraft:deepslate_tile_slab, minecraft:cobbled_deepslate, minecraft:waxed_cut_copper_slab, minecraft:cut_copper_slab, minecraft:potted_azalea_bush, minecraft:magenta_candle, minecraft:deepslate_gold_ore, minecraft:deepslate_tile_wall, minecraft:waxed_exposed_copper, minecraft:deepslate_bricks, minecraft:oxidized_cut_copper, minecraft:raw_copper_block, minecraft:medium_amethyst_bud, minecraft:lime_candle, minecraft:deepslate, minecraft:rooted_dirt, minecraft:copper_ore, minecraft:cobbled_deepslate_slab]>

And some others, but let's do these first.

How do I fix this? I assume that we need to add the colors? By the way I assume that this is done manually right now? I might have a way to automate this by using images of the blocks that are automatically retreived and then calculating the average color of the block.

piegamesde commented 3 years ago

The block colors are automatically generated by the regenerate task from the ./BlockMap-internal/src/main/resources/block-color-instructions-1_17.json file. The file is not super easy to understand (and only a tiny bit documented), but essentially:

Biome colors (if new biomes were added in the update):

Let me know if anything is unclear; this is a great way to test the current state of the documentation.

Also I just noticed, the file sadly has mixed tabs and spaces indentation. Please auto-convert it to one of the other in a separate commit before making actual changes. Sorry for the inconvenience.

uncovery commented 3 years ago

So to understand this from the conceptual side, I need to go through the list in my errors and add those manually to the json file? If so, I'd love to help write a script that automates this process as much as possible to at least create a draft that then could be edited.

piegamesde commented 3 years ago

Pretty much, yes. Actually the whole resource generation thing (plus the unit tests telling you which blocks where added) is the automation of the process, as far as I could take it. I don't see any openings to take this further but if you encounter some repetitive thing that could be automated, let me know.

The process is a bit tedious on updates that add many blocks, but if you got some macros on your editor and are good at copy-pasting, it doesn't actually take that long.

uncovery commented 3 years ago

The thought I had was that the system could look if there is an existing texture that matches the new blocks and automatically creates a sample JSON to be added to the existing one.

piegamesde commented 3 years ago

So the most interesting test for now is de.piegames.blockmap.ColorMapTest > testBlockNames[MC_1_17, DEFAULT], because it will give us a list of all missing blocks. Generally, let's assess all the information that can or cannot be gathered automatically:

Apart from maybe something with texture names, I do not see any hard information to be gained from Minecraft that could benefit us. A possible way forward regarding automation could be to create a list of heuristics that takes off work:

Btw I've had a look at the test results and the ones concerning the update are testBlockNames[MC_1_17, DEFAULT] and testDetailed[MC_1_17, DEFAULT]. testDebugWorld fails for some reason, as well as some for older Minecraft versions (?!) and color maps. I'll have to take a closer look at these…