2% of the generated chunk is not `minecraft:full`

[KokeCacao]

Hi,

I generated 1 very large world with Chunky, but noticed about 2% of the chunks are not in minecraft:full state. My sample consists of 640+ region files and 9845 out of 642376 chunks are not in minecraft:full state. I expect all chunks generated by Chunky to have minecraft:full state. These errors look uniformly distributed.

[pop4959]

Can you provide more details on this issue you are describing?

For the most part I would not expect there to be an issue with this, as chunky waits for full status before generating more chunks. If the chunks are not brought to full chunky stalls to wait for them, so you shouldn't ever get into such a state where you have thousands of them. I'm more inclined to believe that this could be due to user error or a server crash, etc.

Have you tried re-running chunky to ensure that all chunks are correctly brought to full? What platform / version / chunky version are you using? Mods/plugins? What are you using to determine whether the chunks are full or not? Can you show this distribution you are seeing?

It may be convenient to continue discussing this on our Discord server.

[KokeCacao]

@pop4959 Thank you for your reply. Here are the details about my last generation. My method isn't standard so I plan to re-run the experiment when I got time to provide you with more info.

I generated the chunk about 2 months ago using the following code

  @EventHandler(priority = EventPriority.LOWEST, ignoreCancelled = false)
  public void onServerLoad(ServerLoadEvent event) {
    if (config.generateChunks) {
      // Chunky is 140 chunks per second at max on my laptop with CPU performance mode
      // [01:40:37 INFO]: [Chunky] Task running for world. Processed: 92402 chunks (1.00%), ETA: 19:37:49, Rate: 129.5 cps, Current: 161, -150
      // Full generation takes 81.1GB storage, will have 9,247,681 chunks
      Bukkit.dispatchCommand(Bukkit.getConsoleSender(), String.format("chunky shape %s", config.shape));
      Bukkit.dispatchCommand(Bukkit.getConsoleSender(), String.format("chunky radius %d", config.radius));
      Bukkit.dispatchCommand(Bukkit.getConsoleSender(), "chunky start");
    }

  }

where radius is set to 24320 and shape is set to circle. Chunky version is 1.4.10. The server is a paper server running 1.21.1 (ubuntu Linux) and there are no other plugins (other than above code).

I do remember killing the server <5 times during the chunk generation. I can't recall whether I have canceled the generation. But since the full generation takes 19 hours and 2% failing rate means about 22.8 minutes are wasted. The number doesn't make sense to me even if I assume I did not cancel them before killing the server.

I use my own NBT reader to read the states of chunks. Those 2% failure shows all kinds of chunk states such as 'lighting', 'caver' at random but often clustered together.

I will put more information on this issue when I have time to run some other tests. My discord handle is Koke_Cacao.

[pop4959]

Thanks for the extra details. I will close this for now until there is confirmation / additional details on this since at the moment this still isn't really anything actionable.

Considering you are using your own NBT reader, are you certain you are reading the same chunks chunky was generating, or all chunks in the world? I find it likely that if you are simply reading over all regions, you will have some proto-chunks on the edges that are not part of the region generated by chunky as well. This is normal.

If you are certain that after running chunky you are seeing missing full chunks, and this is easily and consistently reproducible, please share your findings and method of testing and I will confirm & re-open this issue.

[KokeCacao]

@pop4959 With the same setting above (changed server.properties to have a fixed level-seed=19) here is what I did for my second experiment:

1. delete the world and Chunky configuration directory
2. start the server (using jdk-21, since jdk-23 seem to have problems)
3. in the command prompt, run chunky shape circle, chunky radius 4000, chunky start
4. after seeing [Chunky] Task finished for world., I killed the server
5. then run my custom script to check chunk integrity

Here is some partial output of my script

hunk 31, 0 is not full: minecraft:structure_startsned:  94%|██████████████████████████████████████████████████████████████████▋    | 233/248 [00:29<00:01,  9.81it/s]
Region (6, -4) has partial data at (31, 0)
Chunk 31, 1 is not full: minecraft:structure_starts
Region (6, -4) has partial data at (31, 1)
Chunk 31, 2 is not full: minecraft:structure_starts
Region (6, -4) has partial data at (31, 2)
Chunk 31, 3 is not full: minecraft:structure_starts
Region (6, -4) has partial data at (31, 3)
Chunk 31, 4 is not full: minecraft:structure_starts
Region (6, -4) has partial data at (31, 4)
Chunk 31, 5 is not full: minecraft:structure_starts
Region (6, -4) has partial data at (31, 5)
Chunk 31, 6 is not full: minecraft:biomes
Region (6, -4) has partial data at (31, 6)
Chunk 31, 7 is not full: minecraft:biomes
Region (6, -4) has partial data at (31, 7)
Chunk 31, 8 is not full: minecraft:biomes
Region (6, -4) has partial data at (31, 8)
Chunk 31, 9 is not full: minecraft:carvers
Region (6, -4) has partial data at (31, 9)
Chunk 31, 10 is not full: minecraft:carvers
Region (6, -4) has partial data at (31, 10)
Chunk 31, 11 is not full: minecraft:initialize_light
Region (6, -4) has partial data at (31, 11)
Chunk 31, 12 is not full: minecraft:initialize_light
Region (6, -4) has partial data at (31, 12)
Chunk 31, 13 is not full: minecraft:initialize_light
Region (6, -4) has partial data at (31, 13)
Cleaned: [22528]                                                                                                                                                     
Total: [219685]

My full code is multi-processing therefore quiet large to share, but the following snippets are the gist of it.

def process_region(args) -> Tuple[int, int]:
    region, dir_path_str, dry = args
    dir_path = pathlib.Path(dir_path_str)
    cleaned = 0
    total = 0
    mca = Mca(dir_path / f"r.{region[0]}.{region[1]}.mca")
    for x in range(32):
        for z in range(32):
            if not mca.data_exists(x, z):
                continue
            total += 1
            if not mca.data_full(x, z):
                print(f"Region {region} has partial data at ({x}, {z})")
                cleaned += 1
                if not dry:
                    mca.delete_data(x, z)
    return cleaned, total

    def data_full(self, x: int, z: int) -> bool:
        payload = self.get_data(x, z)
        if payload is None:
            return False
        nbt = Compound.parse(io.BytesIO(payload))
        status = str(nbt['']['Status'])
        if status not in ['minecraft:full', 'full']:
            print(f"Chunk {x}, {z} is not full: {status}")
            return False
        return True

I think the experiment confirms the problem. The seed is 19 on 1.21.1. And some of the incorrect chunk information are in my output log. I think you have all you need to reproduce this issue.

[pop4959]

I'm not able to reproduce this. I'm assuming your code is including proto-chunks that weren't in the selection you made with chunky, in which case this is very much expected and works as intended.

[KokeCacao]

I'm not able to reproduce this. I'm assuming your code is including proto-chunks that weren't in the selection you made with chunky, in which case this is very much expected and works as intended.

@pop4959 Does the following method filter out the non-chunky generated chunks?

Method:

1. delete the world, start the server
2. close the server, and record all chunks pre-generated by paper
3. start the server again, use Chunky
4. chunky-generated chunks = all chunks - recorded chunks

[pop4959]

I ran your reproduction steps and all chunks in the circle were at full status. Again, I think you are confused since proto-chunks for adjacent chunks required for generation (as part of the chunk pyramid which Mojang calls it) will not be full.

[KokeCacao]

That makes sense. It also explains the radius-dependent proto-chunks-ratio that I have observed. Essentially the ratio is the circumference area divided by the total area, converging to zero as radius approaches infinity.

[pop4959]

Yep exactly. I noticed this in your log as it looks like that region is on the perimeter of your circle.