Implement binary marshal/unmarshal

[jessecarl]

Implement the binary marshal and unmarshal methods. In order to send, receive, and store efficiently, a custom binary format would be useful. It would especially be useful for communicating with clients in other languages. It should be able to be parsed quickly in most languages.

[jessecarl]

Consider the following:

• Some of the properties have a fixed size already, so those are predictable
  • ID is always 16 bytes
  • Status could be represented in as little as 2 bits
  • Since (in the go standard package time.Time.MarshalBinary() implementation) is 15 bytes
• The FriendlyName is (for now) flexible in length.
  • Pick a practical limit (which would need to be enforced elsewhere as well) for the length
  • Store the length of the FriendlyName in the message (may impact the practical limit)
• Like in the time.Time implementation, a version frame would be beneficial
• Is it a better use of space/time to have a minimum of 1 byte for any given frame or to use masks and possibly shifting?
  • Status is already implemented to use the whole byte for now (this could still be shifted or masked before calling UnmarshalBinary()
• The time.Time implementation is nice, but are we willing to rely on it not changing? It would need to remain stable for other language implementations to work.
• Since time.Time is big-endian, and most network protocols are as well, we should probably be big-endian in our implementation.

[jessecarl]

Need to consider the testing strategy we want to employ. I'm inclined to emulate the strategy employed by the time package: test Marshal to Unmarshal, test known bad Marshal, and test bad Unmarshal.

[jessecarl]

Let's consider using the following for the magic numbers in our format here: 0x90,0xe9 I'm sure I'm not the only one to use two consecutive Fibonacci numbers like 144,233, but I couldn't think of anything better.

[jessecarl]

My current thinking:

• 0:1 -- the magic bytes 0x90 0xe9
• 2 -- the version 0x00 for the first version
• 3 -- length of FriendlyName (max of 212 to fit the whole message in 256 bytes?)
• 4 -- XOR of the header so far
• 5:7 -- padding or offset
• 8:23 -- ID
• 24 -- Status
• 25:39 -- Since (using time.Time.MarshalBinary)
• 40:251 -- FriendlyName (utf8 encoded)
• 252:255 -- Adler-32 checksum of message so far (big-endian)

[jessecarl]

This is probably more costly than a naive implementation. The addition of magic bytes provides a quick check that the message is supposed to be a Resource. The version, which may be split in order to provide distinct actions (New, Save, Fetch, etc.) allows a decoder to know how to handle the rest of the message. Put the length of the FriendlyName in the header because it is the only variable-length property of the Resource. Do a simple XOR checksum of the header so far for a quick sanity check. Add some padding to the end of the header as another sanity check, allowing the message itself to start after exactly eight bytes. After the message data, add the Adler-32 checksum for a final sanity check and a maximum message size of 256 bytes.

Again, this is probably overkill for most applications, but the binary format is most likely going to be used on less reliable hardware and networks in embedded devices.

[jessecarl]

Instead of the Adler-32 checksum, which is not ideal for small messages, use CRC32 instead. With a message this small, speed shouldn't be much of an issue.

[jessecarl]

Step back a second.

We'll likely implement an adapter for the formats that require all this extra stuff. I'm going to pull this back again.

A second proposal:

• 0:1 – magic bytes
• 2 – version
• 3 – FriendlyName length
• 4:19 – ID
• 20 – Status
• 21:35 – Since
• 36: – FriendlyName utf8 bytes

I think this should provide a compact, easy to parse message.

[jessecarl]

Testing contract:

For MarshalBinary (independent of UnmarshalBinary):

• magic bytes are correct
• version byte exists and matches the current version
• length exists and matches the length of FriendlyName
• length of byte slice matches the expected length 4 + 16 + 1 + 15 + len(FriendlyName)
• errors when properties are invalid (none yet)

For UnmarshalBinary (independent of MarshalBinary):

• magic bytes are correct
• version byte exists and matches an expected version

For MarshalBinary together with UnmarshalBinary:

• quick test
• edge cases:
  • zero-value
  • out of range status

[jessecarl]

Tests still need to be added to verify a few edge cases and invalid properties.

[jessecarl]

I completed a few additional test cases for bad data.