Message Splitting

[ustulation]

Upper layer might give a big message to Crust. Since currently we have only one TCP connection to the peer so there's single streaming, Crust message sends are atomic. Once begun a partially sent message will be completely sent, no matter its priority, before the next message to send is evaluated. This causes blocking of other (and maybe higher priority) messages until the completion of the current message send.

To prevent this:

0. Crust will as usual accept the message via the send API along with the priroity
1. Crust will analyse if the message is > 1MiB in size. If not queue it in appropriate priority queue and send it (this is similar to what happens just now)
2. If it's > 1MiB then chunk it into multiple parts such that serialised payload of the upper layer message each part is MAX of 1MiB (encryption and crust message headers might take it beyond this but will do so by only a slight margin). Queue all these in the appropriate priority queue and send them. This way such parts can be interleaved with other messages of higher priority in the single stream we have. Reconstruct the entire message (if it's identified as a chunked message) before passing it to the upper layer.
3. Keep messages around until an expiry time. This expiry time will be 300secs. If a split message cannot be reconstructed within this time all the split parts Rxd for the message are dropped. This is so that peer cannot spam our containers
4. Also have an upper max capacity for the split-message-recontruction buffer - 200MiB for all the values + keys combined. If it exceeds the capacity drop all the message parts for the message which was least recently touched in the map - so a TLRU

We can use the following enum to denote a whole message unit vs a chunked message unit:

enum CrustMessage {
    ...,
    ChunkedMsgUnit {
        whole_msg_uid: [u8; 32], // use either a Digest or some UID (allowing upper
                                 // layers to send duplicates if they want)
        complete_msg_len: u32,
        current_chunk: Vec<u8>,
    },
    CompleteMsgUnit(Vec<u8>),
    ...
}

With this the rx should have the read buffer something similar to:

chunked_msgs_read_buf: TLRU<[u8; 32], Vec<u8>>, // 300secs, total size of all the
                                                // values + keys < 200MiB
whole_msgs_read_buf: Vec<Vec<u8>>,

[anderspitman]

This seems like a lot of complexity vs using a multiplexing layer like yamux as suggested in #1135.