Begin implementing Non-consensus node RFC

[imeckler]

@deepthiskumar to make a list of the separate pieces that need to be implemented, then we will piece it out.

[imeckler]

@psteckler to do the gossip net part of this

[deepthiskumar]

Things to do:

• [ ] A Library for block headers, to generate them from blocks (Deepthi)
• [ ] A Library for Non-consensus node state
  1. Block header at the best tip
  2. Block header at the some block height
  3. Account Merkle paths
• [ ] Non-consensus node functionality
  1. Rpc request and responses for the data required by non-consensus nodes
  2. Verify the block header of best tip, root and that the best tip extends the root
  3. Select block headers (Consensus.select)
• [ ] Gossip net layers (Paul)
• [ ] A separate daemon for the non-consensus node

[imeckler]

We need to replicate some computations and some networking in the browser node

Computation

Have snarky-independent versions of

• Public_key.t and signing
• Transaction.t
• Account.t
• Protocol_state.t and state proof verification
• Receipt chain hash and receipt proof verification
• Merkle path verification (can be easily rewritten as a function)
• poseidon hash function (already independent in the sponge library)

Additional functionality

• Tracking finality of various facts. This involves maintaining a tree of protocols states/state hashes.
• List of accounts and transactions that a node is interested in.

Networking

• a gossip message will be

  type gossip_message =
  | New_state of Protocol_state.t * Proof.t

• Use pub/sub libp2p mechanism to make a "non-consensus messages" gossip net layer
• Have libp2p in the browser: One option is to compile libp2p from go to WASM a la 0x. Another is to use js-libp2p. @cmr do you know if js-libp2p is suitable? There was some worry (I'm not sure from whom originally) that the js-libp2p library couldn't communicate with nodes running native libp2p gossip nets.
• Have a function in the browser for getting a pipe of messages on the "non-consensus layer".
• Have a function in the browser for getting a message to the gossip net.
• RPC for ancestor proof
• RPC for getting Merkle paths of accounts at a given state hash.

[enolan]

For fetching the current and "final" account states we need to either make syncable_ledger work on sparse ledgers or write new separate code to fetch the minimal updates necessary. Getting the entire Merkle proof for each new block is too much wasted network traffic. With a ledger depth of 20, that's 20 curve points fetched per block, or ~2k. Twice that since we want the status in two different blocks. That or we need to implement the improved Merkle storage system. I favor option 1.

[psteckler]

We need parameterized versions of Random_oracle and Transaction_union_payload, which are mentioned in Receipt.Chain_hash.

[psteckler]

Also Transaction_union_tag.

[bkase]

We want to prioritize some of the JS client sdk, so if we could prioritize ripping out the graph reachable from these roots it would be ideal:

• Any types used for generating keypairs
• Any types used for creating any sort of transactions
• Any types used for signing transactions

[psteckler]

graph reachable from these roots

the first three items listed above I believe are included in that graph:

- Public_key.t and signing
- Transaction.t
- Account.t

and so far, I've been working on items related to those.

[psteckler]

@bkase : Nathan suggests using dune virtual libraries instead of functors. I'm trying that now, and if it works, no code changes will be needed (and we can undo the bits I've done so far).

[bkase]

oh wow that's nice!

[psteckler]

New plan: use conditional compilation to open Snark_params or some stripped-down module for nonconsensus nodes. Also use conditional compilation to include snarky-dependent code or not. Use consensus_mechanism parameter, which already exists, for that purpose.

Virtual libraries would be nice, but they require writing out interfaces, which we don't have for Snark_params (and other points of snarky contact).

[psteckler]

Stale.