search

NethermindEth / Paprika

A custom storage engine of Nethermind, benefiting from the alignment of the underlying data structure with the layout of State & Storage trees of Ethereum.
GNU Lesser General Public License v3.0
91 stars 14 forks source link

Reduce disk bloat on Windows #265

Open Scooletz opened 8 months ago

Scooletz commented 8 months ago

Use the section API NtCreateSectionFunc to reduce the disk size on Windows. For inspiration take a look at the usage in LMDB. Ensure that on Windows Paprika allocated only as much as needed, not the whole file upfront.

https://github.com/LMDB/lmdb/blob/b8e54b4c31378932b69f1298972de54a565185b1/libraries/liblmdb/mdb.c#L46-L77

One more from

https://devblogs.microsoft.com/oldnewthing/20150130-00/?p=44793

ssonthal commented 2 months ago

Hi @Scooletz, are we focusing on PagedDB block (which uses the memory-mapped files) here?

Scooletz commented 2 months ago

Yes sir! @shubham-sonthalia

ssonthal commented 2 months ago

At the below line, we are initializing the memory mapped file. Do we have to come up with a way to divide the max_size of the Paprika DB into equal partitions (like 5 chunks) and start with 1/5th of the max DB space? or we want it to be dynamic?

https://github.com/shubham-sonthalia/Paprika/blob/666f37225fa14e1adf0d6970f1d911aa8b7b1a49/src/Paprika/Store/PageManagers/MemoryMappedPageManager.cs#L44

image
Scooletz commented 2 months ago

We could be dynamic, but dynamism can come with a price. If we map it separately, there will be different pointers for example.