search

deephacks / lmdbjni

LMDB for Java
Apache License 2.0
204 stars 28 forks source link

LMDB on ramdisk #89

Closed lfoppiano closed 5 years ago

lfoppiano commented 6 years ago

Hi, since several months I have deployed a java application (http://github.com/kermitt2/nerd) using LMDB on a virtual machine with an allocated ramdisk to host the application and the lmdb storage (using tmpfs). The application usual business is to read from LMDB (it's writing only if the databases are to be created).

Until recently everything seemed to work fine, however when we added a new relatively big database the application ended up being killed by the operating system in the effort to free some memory. The more data are processed the more the memory is filled in.

The problem seems to be due to the ramdisk itelf: 

7ce320000000-7cfc20000000 r--s 00000000 00:26 2459                       /ramfs/nerd/data/db/db-en/entityEmbeddings/data.mdb
Size:           104857600 kB
Rss:             1255680 kB
Pss:             1255680 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   1255680 kB    <---
Referenced:      1255680 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd sh mr mw me ms sd

comparing the same application deployed on a normal disk: 

7ca4fc000000-7cbdfc000000 r--s 00000000 fd:00 11154951                   /data/workspace/shared/nerd-data/db/db-en/entityEmbeddings/data.mdb
Size:           104857600 kB
Rss:              838124 kB
Pss:              838124 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:    838124 kB       <----
Private_Dirty:         0 kB
Referenced:       764872 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd sh mr mw me ms sd

According to http://lmdb.readthedocs.io/en/release/#memory-usage the allocated pages needs to be clean for the SO to be able to claim them.

I'm not sure I've understand why we do have this behaviour. Is there a reason? Is there a way to run LMDB on a ramdisk? Any special configuration that could be considered?

Thank you in advance Luca

lfoppiano commented 5 years ago

I think this issue can be closed.

In summary is never a good idea to use lmdb with a ramdisk.