Integrate pollinate - Githubissues

SUMMARY: Integrate pollinate lib to replace DEKU networking & gossip layer

MOTIVATION: Have a better networking layer in DEKU SCENARII:

Integrate pollinate:

Must have UDP networking
Should have observable gossip even if not perfect

DETAILED DESIGN:

@gsebil08 and @AngryStitch 's proposal:

Pollinate offers a way to receive and send messages over the network (via dm or gossip).

In order to isolate Deku from future Pollinate developments and to keep Deku as independant from Pollinate as possible, we offer to code a PROXY_POLLINATE in this way:

module type PROXY_POLLINATE = sig 
  val of_deku : PrenodeMessage.t -> PollinateMessage.t
  val of_pollinate : PollinateMessage.t -> PrenodeMessage.t 

  (* Transforms Deku messages into Pollinate messages. This is where the true dependency on Pollinate happens.   *)
  val handle_outgoing_messages : Pollinate.outbox -> PrenodeMessage.t list -> Pollinate.outbox

  (* Dispatches Pollinate messages to Deku  (different prenodes) *)
  val handle_ingoing_messages : Pollinate.inbox -> Pollinate.inbox * PrenodeMessage.t
end

A PrenodeMessage.t would look like:

PrenodeMessage.t = {
  category: category;  (* Consensus, Operation, ConsensusGovernance, ... *) 
  subcategory: string; (* Proposal, Precommit, Withdraw, AddValidator, ... *)
  payload: bytes;
  sender: Address.t;
  recipient: Address.t option;
 [...]
}

PrenodeMessages will be dispatched according to category to different prenodes. This will be the common interface to every prenode (ConsensusPrenode, ConsensusGovernancePrenode, …):

module type PRENODE = sig 
  type t

  val process_message : Message.t -> t -> t * Message.t list

  val tick : Timestep.t -> t -> t * Message.t list 
end

Those prenodes will be responsible for parts of Deku: Consensus, ConsensusGovernance, BlockOperation (block by hash, block level, ...), Sync (snapshot, ...), ChainOperation (Withdraw, ...), ...

There will be the same number of category & prenodes.

Subcategory is the possible operation inside the category, this is directly linked to the flows.ml operation.

Plus side:

we could change the networking layer AKA Pollinate and we'll only need to rewrite parts of PROXY_POLLINATE
by creating several prenodes, we split different Deku logics into proper categories (readibility, modularity, …)
flows.ml will cease to exist and be split into different and insulated prenodes, which will make testing and understanding easier
common prenode signature for all subparts of Deku
specific messages in Deku that can easily be forged (testing) or re-wrapped (future evolutions)

ALTERNATIVES: TODO

DRAWBACKS:

It will not be a small PR, there will be quite some work.

However, we should do this incrementally:

Add PROXY_POLLINATE
Add PrenodeMessage and PRENODE signature
Add prenode by prenode until everything is out of flows.ml and http endpoints are gone

RELATED PRs:

marigold-dev / deku

Integrate pollinate #567