search

NethermindEth / Paprika

A custom storage engine of Nethermind, benefiting from the alignment of the underlying data structure with the layout of State & Storage trees of Ethereum.
GNU Lesser General Public License v3.0
89 stars 11 forks source link

MMAP makes machine swap a lot #383

Open Scooletz opened 2 weeks ago

Scooletz commented 2 weeks ago

On Windows, memory mapped file occasionally makes a computer swap a lot and makes Paprika behave terribly ppor. With a lot of memory committed, when the size of the mmaped region breaches the space on system disk, it starts to stall. Investigate this behavior and provide a way to fix it. This happens in a scenario of an import where mmaped file is used without any write or flushes and FlushViewOfAFile is called only at the end to be followed by FlushFileBuffers. Still, according to my understanding system should flush what is needed on it's own so that the swap is not affected.

shubham-sonthalia commented 6 days ago

The first solution that can help here is related to dynamically allocating memory mapped files using NtCreateSection APIs. This will reduce the size of the memory mapped files to be written to disk and saving a lot of RAM.

Secondly, during imports, we can try writing calling FlushViewOfFile and FlushFileBuffers after every 10 MB (size-wise) or 5 seconds (duration-wise) instead of doing it all at the end.

Let me know what you think.