search

PowerDNS / lightningstream

Lightning Stream syncs LMDB databases through S3 buckets between multiple servers, including PowerDNS Authoritative server 4.8+ LMDBs
https://doc.powerdns.com/lightningstream/
MIT License
28 stars 16 forks source link

Newer LMDB binary format are not supported #72

Open tbaumann opened 7 months ago

tbaumann commented 7 months ago

When testing lightningstream with my own project written in JS (See https://github.com/kriszyp/lmdb-js/issues/267) I noticed that the files generated are not readable by lightningstream.

Apparently the LMDB libs were updated and new binary formats introduced. And lmdb-go is still using the old one. (https://github.com/kriszyp/lmdb-js/issues/273) Thankfully lmdb-js can fall-back to v1, but I wonder if it would be good to go with the new format here.

I guess this is more of a lmdb-go issue. Perhaps I should raise it there?

PS: In the first issue I posted I posted some code. Do you see any serious issues with it? It seems to work, but I don't have the feeling that lightningstream is yet much used outside the pdns project...

wojas commented 7 months ago

Thanks for reporting!. It looks like LS is still on 1.9.0 of github.com/PowerDNS/lmdb-go. We need to update that to the latest 1.9.2.

wojas commented 7 months ago

Giving it some more thought, I would actually be surprised that an LMDB patch release (from 0.9.28 to 0.9.31) would introduce an incompatible change to the database format.

None of the changelog entries appear to suggest any such change:

LMDB 0.9.31 Release (2023/07/10)
        ITS#8447 - Fix cursor_put(MDB_CURRENT) on DUPSORT DB with different sized data

LMDB 0.9.30 Release (2023/02/08)
        ITS#9806 - LMDB page_split: key threshold depends on page size
        ITS#9916 - avoid gcc optimization bug on sparc64 linux
        ITS#9919 - Mark infrequently used functions as cold
        ITS#9723 - clear C_EOF on cursor with MDB_FIRST_DUP
        ITS#9030 - Use sys/cachectl.h rather than asm/cachectl.h on mips

LMDB 0.9.29 Release (2021/03/16)
        ITS#9461 refix ITS#9376
        ITS#9500 fix regression from ITS#8662

I am not sure what you mean by a v1 and v2 format, this sounds like an lmdb-js thing, not an LMDB thing.

Habbie commented 7 months ago

https://github.com/kriszyp/lmdb-js/tree/dd08273cc5fa238eb86186f4b64506ee56470a51/dependencies - apparently v1 refers to the 'common' 0.9 lmdb branch, and 'not v1' refers to a copy or fork of https://github.com/LMDB/lmdb/tree/mdb.master3

tbaumann commented 7 months ago

It seems to basically come down to two different versions of the lmdb lib. https://github.com/kriszyp/lmdb-js/blob/master/dependencies/lmdb-data-v1/libraries/liblmdb/CHANGES (v0.9.29) And https://github.com/kriszyp/lmdb-js/blob/master/dependencies/lmdb/libraries/liblmdb/lmdb.h which looks like v0.9.90

PS: Ah, thanks Habbie. Looks like you got more context while I was writing