Implement BLOCK protocol message and FlatBuffers builder

[aesedepece]

Context

Inventory exchange protocol

The inventory exchange protocol describes the interaction between nodes in order to synchronize themselves with the network and reestablish the full blockchain (in terms of blocks and transactions).

Inventory exchanges envision 2 main scenarios:

1. Block Download, for nodes in need of synchronizing their blockchains by requesting and downloading blocks.

2. Inventory Broadcasting, for nodes willing to broadcast transactional information, such as blocks and transactions.

Block Download

Block Download is performed by nodes that need to catch up with other nodes by downloading missing blocks in order to reconstruct their blockchains. For example, this is the case of nodes connecting to the blockchain for the first time or after a significant amount of downtime.

Starting from the block #0 (the hardcoded genesis block), the nodes need to validate all blocks up to the current tip of the blockchain. Therefore, in case of missing blocks, a local node will initiate the following process of synchronization with its outbound peers:

1. The local node will have already exchanged version messages with its remote outbound peers. Those version messages contain the last epoch known to others peers, i.e. the local peer can already compare how many blocks they each have and identify how many are missing.

2. The local node will send a get_blocks message to all its outbound nodes (after successful handshake protocol). These messages contain the hash of the top block of the local blockchain.

3. Remote peers will reply by sending another get_blocks message containing the hash of their top block of their respective blockchains.

4. The peer with the longest blockchain will identify which blocks are required by the other peer in order to allow it to synchronize to its blockchain. The peer will select up to the first consecutive 500 blocks and it will transmit their hashes using an inv (inventory) message.

5. After identifying which blocks are missing (it may already have some of them), the node may request them by using a get_data message, containing the hashes of the needed blocks. This message will help the node to catch up with the current full blockchain.

6. After receiving the get_data message, the peer sends the requested blocks individually by using block messages. The following diagram depicts the previously described process under the assumption that the peer with the longest blockchain is NodeB (step 4).

         NodeA                            NodeB
           +                                +
           |           GET_BLOCKS           |
           +------------------------------->+
           |           GET_BLOCKS           |
           +<-------------------------------+
           |              INV               |
           +<-------------------------------+
           |                                |
           |                                |
           |            GET_DATA            |
           +------------------------------->+
           |             BLOCK              |
           +<-------------------------------+
           |             BLOCK              |
           +<-------------------------------+
           |             BLOCK              |
           +<-------------------------------+
           |                                |
           +                                +

Inventory Broadcasting

Similarly to the previously described process of synchronization, any node may contribute to the synchronization of their outbound peers by advertising inventory objects such as blocks and transactions. Inventory broadcasting is also used in case a node creates transactions or mine blocks.

The inventory broadcasting can be described as the following sequence of steps:

1. A remote node broadcasts its inventory by sending an inv message, containing all the hashes of the advertised inventory objects.

2. After receiving an inv message and filtering the inventory objects that may be missing in the local blockchain, the local node sends a get_data message with the hashes of the needed objects.

3. The remote note receives the get_data message and sends a block or tx message per requested inventory object (identified by a hash).

The following diagram depicts the previous step under the assumption that the local node (NodeA) sends a get_data message requesting 3 blocks and 2 transactions.

         NodeA                            NodeB
           +                                +
           |              INV               |
           +<-------------------------------+
           |                                |
           |           GET_DATA             |
           +------------------------------->+
           |                                |
           |             BLOCK              |
           +<-------------------------------+
           |             BLOCK              |
           +<-------------------------------+
           |             BLOCK              |
           +<-------------------------------+
           |                                |
           +                                +

block message

The block message is used to transmit a single serialized block as a response to a get_data message.

The block message consists of a message header with the BLOCK command and a payload formatted as follows:

Field	Type	Description
header	block_header	Block header, as described in the [Block] section
txn_count	u32	Total number of transactions in the block
txns	tx[]	Block transactions as described in the transaction section

Actionables

• [x] Implement block as a Command in the FlatBuffers schema.
• [x] Implement block as a native data structure
• [x] Implement block and its builder as a WitnetMessage in witnet_data_structures::types::Message.

[mariocao]

After discussing it with @Tommytrg, we found out that it is not possible to define arrays with fixed lengths with Flatbuffers. This affects data structures such as hashes, which, in our case, are limited to 256 bits.

Therefore, we thought about the following workarounds.

Option 1: less safe as the hash is not limited to 256 bits.

table Hash {
    bytes: [ubyte];
}

Option 2: more safe but more verbose

table Hash {
    b0: uint64;
    b1: uint64;
    b2: uint64;
    b3: uint64;
}

Other options: even more verbose

• Use different combinations of built-in scalar types (e.g. 32 fields of ubyte/u8)

[mariocao]

Btw, there was an issue in the Flatbuffers project about that. Unfortunately, it was closed without merging any solution.

[aesedepece]

@mariocao I kind of prefer option 1. We can just limit how many bytes we read from hashes when going from flatbuffer-generated types to our own native types.