Messages and their receipts are intended to be separated from the block header (which is all the block data except for those items) and represented in a merkle tree (see spec). This opens up many network efficiencies, light-client functionality etc. In particular, this will make it reasonable to store a fairly long chain of headers in RAM, leaving the messages/receipts in storage most of the time. Memory pressure is a big operational problem for node operators right now (#2031 #2506 #2987).
As a first step, break the messages and receipts each out into a structure that just contains a flat array. See #1324 which follows on with an merkle tree structure for the messages receipts. Immediate benefits to RAM needs may be gained by just extracting them to an array first.
We'll likely want a FullBlock structure that carries the three together for internal convenience.
Acceptance criteria
The Block data structure references message and receipt lists structures by CID
Chain sync fetches these structures for validation
The message/receipt structures can be loaded from the chain store
Risks + pitfalls
The Block type is widely used, this may touch a lot of code
Some places that actually need to read the messages with either a) need to be passed those messages e.g. in a FullBlock or b) have access to a way to load them
This complicates fetching blocks from peers, and should probably wait for Graphsync integration to land first #3104
anorth
commented
5 years ago
Description
Messages and their receipts are intended to be separated from the block header (which is all the block data except for those items) and represented in a merkle tree (see spec). This opens up many network efficiencies, light-client functionality etc. In particular, this will make it reasonable to store a fairly long chain of headers in RAM, leaving the messages/receipts in storage most of the time. Memory pressure is a big operational problem for node operators right now (#2031 #2506 #2987).
As a first step, break the messages and receipts each out into a structure that just contains a flat array. See #1324 which follows on with an merkle tree structure for the messages receipts. Immediate benefits to RAM needs may be gained by just extracting them to an array first.
We'll likely want a
FullBlockstructure that carries the three together for internal convenience.Acceptance criteria
Risks + pitfalls
FullBlockor b) have access to a way to load themWhere to begin