WIP: Multi-frame format

[cdecker]

Quoting the description from https://github.com/lightningnetwork/lightning-onion/pull/31:

• Use the 4 MSB in the first realm byte as a counter of how many 65 byte payload-chunks are destined for the processing node. This clearly distinguishes them from the current payload and directly gives an indication on how much it has to read/allocate.
• Separate payload parsing from the payload extraction, i.e., the onion processing code just hands back a slice of bytes and a realm that are then passed to external parsers that don't interact with the onion processing at all. Mirroring this, when creating a new onion we just get passed the 4 LSB of the realm, and an opaque slices of bytes that may or may not be TLV encoded, we don't care.
• Given the payload length we can infer how many hops we need to read/write. The format looks like this:

For a payload of length <= 32:

|-------+-------------------+------|
| realm | payload + padding | HMAC |
|-------+-------------------+------|

For payloads > 32 and <= 162 for example:

|-------+--------------+-----------------+-------------------------+------|
| realm | payload[:64] | payload[64:129] | payload[129:] + padding | HMAC |
|-------+--------------+-----------------+-------------------------+------|

In other words we use the realm byte from the first hop to determine the number of hops to read, the first hop still has room for 64 bytes of payload (per-hop-payload + HMAC size - realm byte). Any intermediate hop has the full 65 bytes available. The last hop has 33 bytes of payload available, since we will use its HMAC to pass it on to the next node in the path. Notice that we do not decrypt the payloads after the first since processing the first hop already decrypted all the following hops, including the ones we'll be processing. In addition we get a better use of the limited space that we have available and the entire payload is contiguous in memory and can be passed out to the parser as is, without having to stitch it together.

The implementation is also rather trivial, all we need to do is to pass the payload as a byte slice out during processing, and to get the next onion instead of shifting by 65 bytes and padding with 0s, we shift by 65*nhops and pad that with 0s. So the only thing we really need to do is to have the rightShift, headerWithPadding, and copy not take 65, but 65*nhops as arguments.

[cdecker]

Please consider this PR still work in progress, since I'm still working on the command-line tool so we can create test-vectors for the spec.

[rustyrussell]

Would you believe that the only thing I don't like is the you used the 4 MSB not 4 LSB of byte-formerly-known-as-realm? :)

[cdecker]

Would you believe that the only thing I don't like is the you used the 4 MSB not 4 LSB of byte-formerly-known-as-realm? :)

How do you mean? Currently the format is rather nice when encoding the former-realm as hex: 41 means that we use 4 additional frames to encode a payload of type 1 :-)

[Roasbeef]

Closing in favor of #36.