search

dermesser / leveldb-rs

A reimplementation of LevelDB in Rust (no bindings).
Other
515 stars 60 forks source link

DB uses excessive disk space #34

Closed Sword-Smith closed 1 year ago

Sword-Smith commented 1 year ago

It seems to me that I need to call flush explicitly` to prevent the database disk use from blowing up.

Cf. https://github.com/Neptune-Crypto/neptune-core/issues/14

Sword-Smith commented 1 year ago

It seems to still use excessive disk space even after I added a call to flush. A call to delete_obsolete_files or to start_compaction (not sure which one) fixes it. Is there a way I can force the code to call those cleanup functions more often? Either through an explicit call or by changing some value in Options?

dermesser commented 1 year ago

delete_obsolete_files() should be called every time a memtable is written to disk (https://github.com/dermesser/leveldb-rs/blob/master/src/db_impl.rs#L722), which is fairly often (I think every 2-4 MB). This might point at start_compaction() being more helpful.

start_compaction() in turn is called if a VersionSet thinks it needs compaction, which in turn is determined by what I wrote on the other issue: https://github.com/Neptune-Crypto/neptune-core/issues/14#issuecomment-1635880322

In turn, a Version and thus a VersionSet knows its current size (https://github.com/dermesser/leveldb-rs/blob/master/src/version.rs#L497). It may be useful for use cases like yours to expose the current active size on disk (as well as the total size of the database directory), in addition to a manual compaction trigger. It is not beautiful, but permits a higher level of control over this sort of issue.

dermesser commented 1 year ago

I believe #36 will fix your problem, assuming no other ill effects are introduced. I tested it both with a very limited key space that's written again and again - which results in a small set of L1 files - and a larger key space, which as expected percolates into higher levels.

dermesser commented 1 year ago

On Mon, 2023-07-17 at 01:07 -0700, Thorkil Værge wrote:

One of the machines that I used to test the do-more-compactions branch failed with this error. Is it possible that there are some files or file handles that are not being closed? thread 'main' panicked at 'Could not persist to database.: Status { code: IOError, err: "open (write): IOError: Too many open files (os error 24): /home/thv/.local/share/neptune/alpha/databases/wallet_block_db/000001.dbtmp" }', src/models/state/wallet/rusty_wallet_database.rs:93:14 I only received this by email - but this does seem like a potentially problematic issue. Ironic, given that we now handle much fewer files than before... I have a hard time believing that files are leaking unintentionally, as those should be closed automatically, and the number of open table files (which is the only source of so many open files I can think of at the moment) should be handled fairly tightly by the TableCache implementation. By default a database will open up to ~1014 files, but this is configurable in the Options through max_open_files. If you see this error, I'd first suggest tuning that.

Sword-Smith commented 1 year ago

On Mon, 2023-07-17 at 01:07 -0700, Thorkil Værge wrote: One of the machines that I used to test the do-more-compactions branch failed with this error. Is it possible that there are some files or file handles that are not being closed? thread 'main' panicked at 'Could not persist to database.: Status { code: IOError, err: "open (write): IOError: Too many open files (os error 24): /home/thv/.local/share/neptune/alpha/databases/wallet_block_db/000001.dbtmp" }', src/models/state/wallet/rusty_wallet_database.rs:93:14 I only received this by email - but this does seem like a potentially problematic issue. Ironic, given that we now handle much fewer files than before... I have a hard time believing that files are leaking unintentionally, as those should be closed automatically, and the number of open table files (which is the only source of so many open files I can think of at the moment) should be handled fairly tightly by the TableCache implementation. By default a database will open up to ~1014 files, but this is configurable in the Options through max_open_files. If you see this error, I'd first suggest tuning that.

I think I misread a log and that it's from prior to me trying out your new branch. Should have edited the original note instead of deleting it.

dermesser commented 1 year ago

Don't worry please, I just want to be extra careful with this change. As we've seeen, the specific compaction behavior can have some rather nasty impact...