Choosing a serialization format and library

[sanderman01]

Having a standard format for both networking messages and dynamic pieces of data will eventually be required.

The Game Minecraft uses the NBT format for both local storage and network transfer of data. This is used for a lot of things that need the ability to store some small amount of data somewhere:

• TileEntitites may need to store their state in a NBT.
• ItemStacks use NBT to store information detailing which item they contain, their quantity, health, and any tags that may be attached to the itemstack. The latter tagging mechanism is used for things like enchantments.
• Player Inventory is stored as a list of compound NBT tags.

This mechanism is very powerful because it allows for passing around very dynamic pieces of data, and it is also forward compatible. I feel that I want to take a similar approach for my own game. I intend to use a more common and well-supported serialization format though.

The format and API should be:

• Compact in binary form and low performance overhead. (Some types of messages --like player movement related-- will be sent multiple times per second)
• Dynamic. (Messages are not based on a predefined schema. They are dynamically constructed. This makes it easy to add extra fields to a message later without it affecting existing code.)
• Either human readable, or easy to convert to some other human readable format.
• Relatively simple to work with in code.

Right now I'm leaning towards MsgPack. If anyone has a suggestion and rationale for picking a different option, please discuss.

References: NBT Format MsgPack BSON vs MsgPack discussion

[sanderman01]

Looks like CBOR is also a good candidate. It seems inspired by both JSON and MsgPack, and it looks more 'officially' standardized based on an RFC and all that stuff.

CBOR