Incorrect unicode character handling for chat messages

[Yaodeowo]

Idk is it a server issue or a bot issue, I'm having some issues making the chat parser and I don't know how to fix it. Is it possible to fix it?

Bot Version: 1.20.4 Server Software: Scissors 1.20.4 Library Used: minecraft-protocol 1.50.0 NodeJs Version: v18.17.1

Code: const mc = require('minecraft-protocol'); const bot = mc.createClient({ host: '',
port: 00000, username: '', version: "1.20.4", });

bot.on('system_chat', (packet) => { console.log(JSON.stringify(packet)) }

3 Issues

Invalid Emoji Handling

Only Unicode range U+0001 to U+9999 works correctly. Emojis outside this range are displayed as �

Command: /minecraft:tellraw @a "🌫" Output: {"content":{"type":"string","value":"������"},"isActionBar":false}

Spawnpoint Angle

{"content":{"type":"compound","value":{"italic":{"type":"byte","value":1},"color":{"type":"string","value":"gray"},"with":{"type":"list","value":{"type":"compound","value":[{"insertion":{"type":"string","value":"Username"},"clickEvent":{"type":"compound","value":{"action":{"type":"string","value":"suggest_command"},"value":{"type":"string","value":"/tell Username "}}},"hoverEvent":{"type":"compound","value":{"action":{"type":"string","value":"show_entity"},"contents":{"type":"compound","value":{"type":{"type":"string","value":"minecraft:player"},"id":{"type":"intArray","value":[-304927290,1767587547,-1427566720,-1797465230]},"name":{"type":"string","value":"Username"}}}}},"text":{"type":"string","value":"Username"}},{"with":{"type":"list","value":{"type":"compound","value":[{"":{"type":"int","value":0}},{"":{"type":"int","value":0}},{"":{"type":"int","value":0}},{"":{"type":"float","value":0}},{"":{"type":"string","value":"minecraft:overworld"}},{"insertion":{"type":"string","value":"Username"},"clickEvent":{"type":"compound","value":{"action":{"type":"string","value":"suggest_command"},"value":{"type":"string","value":"/tell Username "}}},"hoverEvent":{"type":"compound","value":{"action":{"type":"string","value":"show_entity"},"contents":{"type":"compound","value":{"type":{"type":"string","value":"minecraft:player"},"id":{"type":"intArray","value":[-304927290,1767587547,-1427566720,-1797465230]},"name":{"type":"string","value":"Username"}}}}},"text":{"type":"string","value":"Username"}}]}},"translate":{"type":"string","value":"commands.spawnpoint.success.single"}}]}},"translate":{"type":"string","value":"chat.type.admin"}}},"isActionBar":false}

Command: /minecraft:spawnpoint @p 0 0 0 0.0 Actual Output: {"":{"type":"float","value":0}} [Username: Set spawn point to 0, 0, 0 [0] in minecraft:overworld for Username] Expected Output: {"text":{"type":"float","value":0.0}}

Command: /minecraft:spawnpoint @p 0 0 0 0.1 Actual Output: {"":{"type":"float","value":0.10000000149011612}} [Username: Set spawn point to 0, 0, 0 [0.10000000149011612] in minecraft:overworld for Username] Expected Output: {"":{"type":"float","value":0.1}}

Empty Text Field "" This issue already report by Parker2991, BF5258

Command: /minecraft:tellraw @a {"translate":"","with":[0.1,1]} Actual Output: {"content":{"type":"compound","value":{"with":{"type":"list","value":{"type":"compound","value":[{"":{"type":"double","value":0.1}},{"":{"type":"byte","value":1}}]}},"translate":{"type":"string","value":""}}},"isActionBar":false}

Command Blocks (e.g., /give or /fill) Fields such as {"":{"type":"string","value":"@"}} appear unexpectedly.

I have test it long time, it just like "text":"" to "":""

I think the patch code shown by BF5258 is a good way to patch BF5258 Issues Link

[extremeheat]

Please make one issue per problem

For the fog emoji issue, after looking into it I can reproduce on Minecraft Java Edition 1.20.6.

In minecraft 1.20+, chat messages are no longer sent as JSON, but as NBT. I looked into this with ChatGPT and the problem lies with our NBT string decoding.

minecraft writes the fog emoji string over the network like this: ed a0 bc ed bc ab

Inside minecraft-protocol/prismarine-nbt we currently decode the NBT strings as UTF-8, but Minecraft is encoding this string as UTF-16. It's a bit weird as most parts of the protocol are UTF-8 so might have to look into this some more. Maybe NBT is still UTF-16? cc @GroobleDierne

If so we can fix by patching prismarine-nbt's shortString type to decode as UTF-16 which will correctly decode as 🌫️

[GroobleDierne]

According to wiki.vg Minecraft uses a u16 prefixed modified UTF-8 string as defined here https://docs.oracle.com/javase/8/docs/api/java/io/DataInput.html#modified-utf-8

[extremeheat]

wiki.vg is not reliable source, have to look at the game code. The encoding above is not valid UTF-8, as the string decoder is falling back to U+FFFD chars which is an error placeholder. It is valid big UTF-16:

ED A0 BC corresponds to the high surrogate D83C in UTF-16.
ED BC AB corresponds to the low surrogate DF2B in UTF-16.

Seems only system_chat command NBT is using UTF-16, other NBT strings are UTF-8 (could be other packets beyond system_chat too). This is a problem as the two encodings are not interchangeable.

This is not easy to fix in prismarine-nbt because the way its protocol spec uses recursion (means we can't just pass down a special flag). Also it seems endianness may be a problem here with Node's Buffer string encoding methods, so that may complicate things further.

[extremeheat]

Actually you may be right.. I'm not familiar with it, but it could be "modified UTF-8" (bad name). Doesn't really seem like UTF-8 at all based on the website description:

"Supplementary characters are represented in the form of surrogate pairs." Only UTF-16 needs them, and I don't understand why it's needed to introduce them at all here. Basically seems like they switch between UTF-8 and UTF-16 based on the character... kind of surprised this was not picked up earlier.

Seems like we'd need to write a custom text encoder/decoder for this

[rom1504]

At this point I really wonder if they on purpose try to make it hard for other people to implement the protocol

[GroobleDierne]

After some tinkering I think I have a working decoder for Java's modified utf8

[extremeheat]

The more annoying thing is that for most network serialization code, the biggest bottleneck is usually text encoding by far. It's expensive to encode/decode unicode like it is to decode Minecraft chunk data as it involves lots of memory operations on a char by char basis. And since strings can get big easily (>1MB), and there's many of them, it's not good idea to do in JS given it's going to create alot of intermediate garbage. We're in Minecraft though so it's not a big deal, but it is something to note.