arkpar
commented
9 months ago This is probably caused by the way memory maps are re-created when growing a file. On linux we try to reserve an 1Gb of address space for the mapping so that the mapping address could be reused. On windows a new mapping is created on each file increase. On linux we also simply leak the old mapping, which would have been fine for 1Gb increases, but not for small increases.
https://github.com/paritytech/parity-db/pull/234 should fix the leak, could you give it a test?
nazar-pc
After many complaints from users of our Substrate-based chain I did some experiments and discovered that with ParityDb application uses a lot of memory on Windows until it runs out of memory, while RocksDb works fine.
I tried both mimalloc allocator and default system allocator with the same result.
Errors typically look something like this (there are other variations, sometimes other components run out of memory first):
Interestingly enough, task manager shows very reasonable memory usage of the app in the process list, while there is a huge total memory usage in Performance tab.
In Sysinternals Process Explorer I see that while the app uses ~3G of "Working Set" memory (similarly to what task manager shows), "Private Bytes" is over 10x more (again, not happening with RocksDb).
Not sure how to narrow it down more yet, so open for suggestions and tests, but this is a pretty big issue that I'd like to help resolve.