Auth: LMDB Backend, when importing 33 GB zone records using pdnsutil the import crashes after 16GB

[fireangel403]

          I have checked importing 33 GB zone records and the import crashes after 16GB 

pdnsutil load-zone example.com example.zone
Zone 'example.com' exists already, replacing contents
Error: putting data: MDB_MAP_FULL: Environment mapsize limit reached

I updated the config file (/etc/powerdns/pdns.conf) to as per the following:

launch=lmdb
lmdb-filename=/var/lib/powerdns/pdns.lmdb
lmdb-sync-mode=nometasync
lmdb-map-size=128000

reuseport=yes
receiver-threads=8
max-cache-entries=14000000

service pdns restart

Same error.

Error: putting data: MDB_MAP_FULL: Environment mapsize limit reached

It seems thatlmdb-map-size=128000has no effect.

ls -alh /var/lib/powerdns/
total 16G
drwxr-xr-x  2 pdns pdns 4.0K Jun  1 19:26 .
drwxr-xr-x 45 root root 4.0K Jun  3 18:52 ..
-rw-------  1 pdns pdns  44K Jun  1 19:26 pdns.lmdb
-rw-------  1 pdns root  16G Jun  9 12:12 pdns.lmdb-1
-rw-------  1 pdns root 8.0K Jun  9 13:01 pdns.lmdb-1-lock
-rw-------  1 pdns pdns 8.0K Jun  9 13:01 pdns.lmdb-lock

PDNS Version: 4.8.0

DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"

Originally posted by @fireangel403 in https://github.com/PowerDNS/pdns/issues/11243#issuecomment-1584541865

[zeha]

Please:

• where did your pdns packages come from
• what is their exact version
• output of uname -a

[fireangel403]

• Packages were installed from the 4.8 repo

cat /etc/apt/sources.list.d/pdns.list
deb [arch=amd64] http://repo.powerdns.com/ubuntu focal-auth-48 main

• version

pdns_control version
4.8.0

• output of uname -a

Linux pdnslmdb 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[phonedph1]

This might be silly, but if you create a few 'garbage' zones before trying the import, does it work? It looks like it's not creating all the shards (which in and of itself might be a different bug...)

[fireangel403]

I kind of worked around the issue by recompiling after editing the files

ext/lmdb-safe/lmdb-safe.hh
modules/lmdbbackend/lmdbbackend.cc

and changing the following lines:

std::shared_ptr<MDBEnv> getMDBEnv(const char* fname, int flags, int mode, uint64_t mapsizeMB=(sizeof(void *)==4) ? 100 : 16000);

declare(suffix, "map-size", "LMDB map size in megabytes", (sizeof(void*) == 4) ? "100" : "16000");

replacing 16000 with 70000

recompile and replace the following installed binary files and library file with the recompiled ones in my server.

/usr/bin/pdns_control
/usr/bin/pdnsutil
/usr/sbin/pdns_server
/usr/lib/x86_64-linux-gnu/pdns/liblmdbbackend.so

[fireangel403]

But that basically means that the value from the config file is not being read properly

[kpfleming]

If this is a bug (which it seems to be), the issue should be left open and the title changed to reflect that.

[fireangel403]

Status changed to open.