Avoid all bit twiddling in consensus storage layer

IntersectMBO / ouroboros-consensus

Implementation of a Consensus Layer for the Ouroboros family of protocols

https://ouroboros-consensus.cardano.intersectmbo.org

Apache License 2.0

31 stars 22 forks source link

Avoid all bit twiddling in consensus storage layer #662

Open edsko opened 4 years ago

edsko commented 4 years ago

The storage layer provides an API for binary streaming, but due to the way we've set things up, this requires the storage layer to do bit twiddling, interpreting the binary data, dropping stuff, adding stuff, changing stuff, all based on raw binary data. This is nasty.

edsko commented 4 years ago

This involves at least:

Changing the on-disk format so that the encoder/decoder for header can directly be applied to however we store the header. In other words, we can't just write the block, because the nested header often uses a different encoder (missing CBOR-in-CBOR tag, missing HFC tag, etc.)
Dropping old versioning protocols that required us to fake a size hint for Byron.

edsko commented 4 years ago

We should get to a point where in principle for binary streaming we could just point the OS at the disk and tell it "go".