search

PowerDNS / pdns

PowerDNS Authoritative, PowerDNS Recursor, dnsdist
https://www.powerdns.com/
GNU General Public License v2.0
3.61k stars 900 forks source link

document LMDB memory needs #7729

Open franklouwers opened 5 years ago

franklouwers commented 5 years ago

Short description

"I solemnly promise that documentation is forthcoming," a wise man once said. This is a gentle reminder to that wise man...

franklouwers commented 5 years ago

Would love some notes on memory usage (with/without dnssec) as well, as the database is in-mem.

hyc commented 5 years ago

^ LMDB is not in-memory. It is a file-based DB like any other persistent DB. It just uses memory-mapping to read data, instead of read() syscalls, and it leverages the OS cache manager instead of having its own cache that admins must configure and tune.

When there is no memory pressure and there are a large number of accesses, the cache may fill with accesses. When there is high memory pressure the cache will relinquish memory for other apps to use, automatically and transparently. Memory usage is completely a non-issue.

ahupowerdns commented 5 years ago

Well.. :-) Except on 32 bit, where we quickly run out of mmap()able address space, so there it is important to know how much file needs to be mapped into memory. But I appreciate the nuance.

hyc commented 5 years ago

Running out of address space is a separate issue, definitely.

On 32 bit you have a choice - operate with a single map, like 64 bit builds do, but be limited to ~2GB address space, or operate in chunks, and have the same overall size limits as a 64 bit build, but slower performance. (For the latter, compile from the git mdb.master branch with -DMDB_VL32).

franklouwers commented 5 years ago

Thanks for the updates. Looking forward to the docs even more now :)

Habbie commented 5 years ago

We have documentation now https://doc.powerdns.com/authoritative/backends/lmdb.html

nothing about memory usage - I can leave this ticket open for a while for that.