search

capocasa / limdb

Fast, in-process key-value store with a table-like interface persisted to disk using lmdb.
MIT License
34 stars 3 forks source link

initDatabase with size=5e9.toInt creates a 5GB database on disk even though the database is empty #4

Closed greenersharp closed 1 month ago

greenersharp commented 1 month ago

I am seeing some odd behavior in this implementation of LMDB on Linux.

When I initDatabase with size=5e9.toInt, this nim implementation immediately creates a 5GB file on disk even though the database is empty.

However, when using LMDB on python or C++ this does not happen. In python and c++ implementations, the database file size begins empty and then slowly grows as i insert data, until reaching the max size. If I only insert 1 gig of data, the database will be 1 gig in size. In other words the size of the database matches the size of the data I have inserted.

This is a known issue with LMDB, on windows, but this shouldn't be happening on Linux. https://stackoverflow.com/questions/33508305/lmdb-maximum-size-of-the-database-for-windows

Any ideas ?

Thank you.

greenersharp commented 1 month ago

sorry just realized the machine I was using had a FAT32 partition.

capocasa commented 1 month ago

Thanks for taking the time to report!

I would have been rather surprised, since this is just a syntactic wrapper on the C version, but never say never. I was suspecting the default configuration parameters, since those are the only real difference, but glad it cleared up.

Feel free to place a brief warning into the docs under known issues as a pull request if you like, to share your hard-earned bit of knowledge!