A custom storage engine of Nethermind, benefiting from the alignment of the underlying data structure with the layout of State & Storage trees of Ethereum.
This PR amends layout of storage one more time. It makes StorageFanOut to be split into levels Level0, Level1, Level2, Level3. This allows to align it better with the Keccak -> uint mapping and use the id that Keccak is mapped to directly, without path appending.
The levels:
Level0 - embedded in RootPage provides an initial fan out of 1024. Both Keccak->uint mapping as well as the actual storage goes through this
Level1 separates Ids from the Storage
Ids provides a fanout of 64 that effectively gives 64k buckets for Keccaks to be mapped to
Storageprovides additional fanout of 1024 consuming another 10 bits of uint contract id.
Level2 consumes next 10 bits providing the fan out of 1024
Level3 has only 2 bits left for addressing (32 - 10 + 10 + 10). It uses it to address the contract in one of the 4 buckets in the page
The crux lies in how the bits of uint id of the contract are consumed. They are consumed from the highest to the lowest, meaning that two consecutive ids like 1U and 2U will have the save Level1 and Level2 and will reside in two different buckets of the same page Level3. This should ease with navigation through the tree but at the same time make the db smaller for smaller networks (less contracts -> smaller number of buckets used on Level0 that spreads across highest 10 bits of uint id).
Ethereum mainnet
Importing Ethereum mainnet with Paprika.Importer shows a great reduction of application time (simpler logic for addressing, better inserts). This should also reduce the reading time as to get to an contract storage slot one need to go through 3 levels of the fanout pages (level0 is at the root) and then start querying. It should be top 5 levels of pages that should be scanned.
This PR amends layout of storage one more time. It makes
StorageFanOut
to be split into levels Level0, Level1, Level2, Level3. This allows to align it better with theKeccak
->uint
mapping and use the id thatKeccak
is mapped to directly, without path appending.The levels:
Level0
- embedded inRootPage
provides an initial fan out of1024
. BothKeccak
->uint
mapping as well as the actual storage goes through thisLevel1
separatesIds
from theStorage
Ids
provides a fanout of64
that effectively gives 64k buckets forKeccak
s to be mapped toStorage
provides additional fanout of1024
consuming another 10 bits ofuint
contract id.Level2
consumes next 10 bits providing the fan out of1024
Level3
has only 2 bits left for addressing (32 - 10 + 10 + 10). It uses it to address the contract in one of the 4 buckets in the pageThe crux lies in how the bits of
uint
id of the contract are consumed. They are consumed from the highest to the lowest, meaning that two consecutive ids like1U
and2U
will have the saveLevel1
andLevel2
and will reside in two different buckets of the same pageLevel3
. This should ease with navigation through the tree but at the same time make the db smaller for smaller networks (less contracts -> smaller number of buckets used on Level0 that spreads across highest 10 bits of uint id).Ethereum mainnet
Importing Ethereum mainnet with
Paprika.Importer
shows a great reduction of application time (simpler logic for addressing, better inserts). This should also reduce the reading time as to get to an contract storage slot one need to go through 3 levels of the fanout pages (level0 is at the root) and then start querying. It should be top 5 levels of pages that should be scanned.Import stats
CLI stats