Issues with blockStates Array

[Paulomart]

I tried to render worlds that have been downloaded with a mineflayer bot and encountered some glitches. Initially I reported these problems to the mcmap project, but its now beliefed that there is an underlinding issue in the chunk implementation of mineflayer.

The issue boils down to a different format when writing the blockStates array in a chunk section.

Mineflayer	Vanilla
value 37191016277606400 33825 0 68719476736 0 2357352929953794 68719575043 4398046511104 5242880 5497558138882 4194304 4398052802624 67108864 2052 4196416 68721641632 2206541545472 2322168557862912 2251799813685248 70368744180739 72130161805361152 14338 2147483712 0 0 0 0 103082360928 0 5497558138884 0 4398046511109 6597069766656 4 4194304 4398046511104 167772160 7881299347898368 7 0 106404249600 0 0 0 0 0 0 0 108089689692438528 370702544327933952 4465995 320357116197 373109799749943296 482272188221620228 4567462 4753871744320 335918783468142592 370596991211667460 271691 7347200 0 111572942428569600 0 0 0 0 0 0 0 105553116266496 98304 4682935510272 335918783473385472 445082306923724800 339341 4789331472736 373109799744700416 407892425625829376 4499820 147450045600 7696581394432 0 0 101475 0 0 0 0 0 0 0 0 3298534883328 401946266742816768 141676 12227849216 158090440594161664 444906385063280640 11661 4677019648 7937099563008 0 0 0 106404249600 373034608490446848 265 0 0 0 0 0 0 0 3 347411456 11659609995476992 444730463202836480 11661 11119470752 364281946177536 0 0 0 0 108195242708041728 0 278211913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 106300440576 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 111464090777419776 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0	value 1229782938247299072 0 35184372088832 144678138029277184 2305895798693429248 1125899906842624 562949953748992 1125899907170304 1125925678743552 145241087984795648 1128236369051648 35193537691648 8707506176 35651584 9007199305269760 577059032657821696 144115325514809344 0 0 0 52789446180864 1125899906842624 327680 1125899907170304 1125925676646400 1125899906842624 1125899906842624 5242880 30182211584 0 858980352 0 0 0 0 0 52789446180864 1296035281371136 188900373233664 1333565278453760 1352291621797888 1333565278126080 1314800279748608 170135370334208 30182211584 0 858980352 0 0 0 0 0 52776561278976 1296035281371136 188900373233664 207665371283456 1352291621797888 207665371283456 1314800279748608 82174440112128 30064771072 0 858980352 0 0 0 0 0 0 12884901888 83347145490432 14151313260544 85654219259904 14151318503424 5281920581632 31943819264 0 0 858980352 664591269888 0 0 0 0 0 12884901888 810490527744 12978501976064 14151313260544 12978507218944 810406641664 0 0 0 808648704 664591269888 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 855638016 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 855638016 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

As you can see there is a length difference between vanilla and mineflayer.

@spoutn1k (maintainer of the mcmap project) investigated this difference further. He pointed out, that the chunk section in vanilla uses 4 bits per block. Mineflayer writes that section with 5 bits per block.

You can see the difference when looking at the two binary values:

Mineflayer (5 bits per block)

0000000010000100001000010000100001000010000100000000000000000000
00000 00010 00010 00010 00010 00010 00010 00010 00010 00000 00000 00000 0000

Vanilla (4 bits per block)

0001000100010001000100010001000100010001000100010000000000000000
0001 0001 0001 0001 0001 0001 0001 0001 0001 0001 0001 0001 0000 0000 0000 0000

In mineflayer, there is a strange 4 bit zero padding added to the start. When cutting this in 5 bit chunks, this shifts all bits to the right. 00010 vs 0001 in vanilla.

If we would drop the 4 bit prefix, it fixes the alignment and the stored values are correct:

Mineflayer (5 bits per block, prefix seperated)

0000000010000100001000010000100001000010000100000000000000000000
0000 00001 00001 00001 00001 00001 00001 00001 00001 00000 00000 00000 00000

---

There has already been some discussion about this on discord. I tried to sum up my findings in this issue. See this message and the following for details: https://discord.com/channels/413438066984747026/743199746876768318/921329942917181470

[Karang]

A solution for the bit size would be to resize the bitarray just before writing the blocks in https://github.com/PrismarineJS/prismarine-provider-anvil/blob/master/src/1.14/chunk.js#L185

writeBlocks(section.data.resizeTo(neededBits(section.palette.length)))

neededBits can be taken from https://github.com/PrismarineJS/prismarine-chunk/blob/master/src/pc/common/neededBits.js

The reason for the difference might be that mineflayer never resize down a bitArray (it can only grows) which is an optimisation for the program while it's running but waste space when you store it to disk.

I have no idea where the padding come from, but try fixing that first.

[nickelpro]

The missing piece of this puzzle is the intermediate operations. What was done to the chunks between loading and storing? Merely performing the load-store with pris-anvil should be isomorphic, no change to the data.

I agree with Karang, we can partially explain the expanded palette as being caused by us not compacting before storage, which implies operations were done that caused the palette to temporarily go from 4->5 bits long.