Heightmap too large for region file

[DaMatrix]

What the title says. I mentioned this in the discord server a while back, then remembered about it just now and decided to report it.

Error

Caused by falling down an infinite hole with random blocks generated on all sides, after about 2 minutes of laggy falling the log was filled with these errors, and new cubes didn't save any more. I'll end this by posting the old chat log about the error.

Thanks!

[DaMatrix]

Still happening. Is there any workaround at all (even if it's just expanding the max size)?

[11:49:31] [File IO Thread/ERROR] [cubicchunks]: Unable to write column (766, 7) java.lang.IllegalArgumentException: Supported entry size range is 0 to 255, but got 280 at cubicchunks.regionlib.region.IntPackedSectorMap.packed(IntPackedSectorMap.java:102) ~[IntPackedSectorMap.class:?] at cubicchunks.regionlib.region.IntPackedSectorMap.setOffsetAndSize(IntPackedSectorMap.java:61) ~[IntPackedSectorMap.class:?] at cubicchunks.regionlib.region.RegionSectorTracker.reserveForKey(RegionSectorTracker.java:49) ~[RegionSectorTracker.class:?] at cubicchunks.regionlib.region.Region.writeValue(Region.java:71) ~[Region.class:?] at cubicchunks.regionlib.SaveSection.lambda$save$0(SaveSection.java:63) ~[SaveSection.class:?] at cubicchunks.regionlib.region.provider.CachedRegionProvider.forRegion(CachedRegionProvider.java:148) ~[CachedRegionProvider.class:?] at cubicchunks.regionlib.region.provider.CachedRegionProvider.forRegion(CachedRegionProvider.java:84) ~[CachedRegionProvider.class:?] at cubicchunks.regionlib.SaveSection.save(SaveSection.java:63) ~[SaveSection.class:?] at cubicchunks.regionlib.impl.SaveCubeColumns.save2d(SaveCubeColumns.java:64) ~[SaveCubeColumns.class:?] at cubicchunks.server.chunkio.RegionCubeIO.func_75814_c(RegionCubeIO.java:189) [RegionCubeIO.class:?] at net.minecraft.world.storage.ThreadedFileIOBase.func_75736_b(SourceFile:37) [bgx.class:?] at net.minecraft.world.storage.ThreadedFileIOBase.run(SourceFile:30) [bgx.class:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

[Barteks2x]

I would suggest modifying RegionLib to use 0xFFFFFFFF header data as "stored externally" flag and if exceeds max size - store in external file

[DaMatrix]

Excuse my ignorance, but where specifically? RegionLib appears a huge tangle of misc. int arrays and bytebuffers. I've set SIZE_BITS in IntPackedSectorMap to 16 (from 8) because that's where the stack trace was pointing, is that really bad?

[Barteks2x]

the limitation really comes from 32 bit integers... one 32 bit int needs to store offset into the region, and the SIZE of an entry in sectors (could use 64-bit but that would double the size of empty region).

Currently this is done by using 8 bits for size and the rest for offset. This probably isn't optimal but this is how vanilla does it.

Changing that value means that you completely break world save compatibility and all hell breaks loose with existing saves.

So what would be the optimal way? I think it can be calculated. Considering max entry size being 2^SIZE_BITS - 1, and the amount of them being fixed 32*32 per region for columns and 16*16*16 for cubes (columns are what we are interested in), and the maximum possible size of "holes" between entries (the regions don't support fragmentation) is 2^SIZE_BITS - 2 (if it was one sector bigger, you would have free space to use), and assuming a very reasonable heuristic that you can't really get all of the holes at that size, and roughly half the total hole size will be there, you get total max region size in sectors for columns of 32*32*(2^SIZE_BITS - 1)*1.5 = 1536*(2^SIZE_BITS - 1).

We also know that SIZE_BITS = 32 - OFFSET_BITS. And the max total region size should be equal to 2^OFFSET_BITS (roughly, it's a good approximation).

So:

1536*(2^SIZE_BITS - 1) = 2^(32 - SIZE_BITS)
for bigger SIZE_BITS we can assume 2^SIZE_BITS - 1 being almost equal to 2^SIZE_BITS
1536*(2^SIZE_BITS) = 2^(32-SIZE_BITS)
2^(32-SIZE_BITS) / 2^SIZE_BITS = 1536
2^(32 - SIZE_BITS*2) = 1536
2^32 = 1536*2^(SIZE_BITS*2)
log2(1536) = 10.58(...) -> 1536 = 2^10.58
2^(32 - 10.58) = 2^(SIZE_BITS*2)
21.41 = SIZE_BITS*2
SIZE_BITS = 10.7

So the optimal amount of bits for size is about 10.7 bits, and considering there aren't going to be THAT many holes in real world, we can round it up to 11. But again that breaks save compatibility.

Instead of changing that you can modify Region class. What I'm going to describe will be a quick hack and not a properly designed solution.

• add isExternal method to RegionEntryLocation, that returns true when offset == -1
• Move RegionEntryLocation.findSectorFor to IKeyIdToSectorMap and make the implementation the default method
• Override it in IntPackedSectorMap, call super and if size exceeds max supported value, return RegionEntryLocation with location -1 and size 0
• in .IntPacketSectorMap.unpackOffset, you check for value OFFSET_MASK and if it's equal, you return -1
• in Region class, this is the "quick and dirty hack" part, in writeValue and readValue part, you check if the location you got isExternal and if so, just read/write to/from separate file in ext subdirectory, named with $REGION_NAME.E$ENTRY_ID.bin

[DaMatrix]

This seems to be fixed now, albeit hackily. Should I close?