search

Level / abstract-level

Abstract class for a lexicographically sorted key-value database.
MIT License
128 stars 8 forks source link

clear without limit #30

Closed ronag closed 2 years ago

ronag commented 2 years ago

rocksdb has a very nice DeleteRange API which is non blocking and would be great for clear. However, there are two problems:

Any chance of making these optional in the level API?

ronag commented 2 years ago

Optionally if we could add a deleteRange API call and discourage clear.

vweevers commented 2 years ago

Why is DeleteRange better? Especially if it does not support these options?

ronag commented 2 years ago

Because it's non blocking. It can apply the delete lazily while applying it to all preceding reads in memory. Also you can do delete range in a batch.

vweevers commented 2 years ago

Not sure I understand what you mean by non blocking.

ronag commented 2 years ago

Not sure I understand what you mean by non blocking.

It doesn't block the thread due to IO.

vweevers commented 2 years ago

From the JS side that doesn't matter. Unless your concern is not having enough threads in the async worker pool (for things other than level)?

ronag commented 2 years ago

I'm actually trying to do all the write operations in the main js thread due to a few different reasons:

In rocksdb and I believe also leveldb writes are non blocking as long as the writing background threads can keep up.

vweevers commented 2 years ago

Not sure if there's anything left to answer here. I'll address some specific points:

Open end is not supported

That simply makes it incompatible with the abstract-level API. Unlike limit, the use cases for ltgt options are (I hope) obvious and these options are a core part of level. So can't make this an optional feature.

Limit is not supported

See https://github.com/Level/abstract-level/issues/38#issuecomment-1264189479 - which was a duplicate issue?

Optionally if we could add a deleteRange API call and discourage clear.

I don't want to add RocksDB-specific features that are a limited subset of existing functionality. Your implementation is free to have additional methods ofc (like deleteRange). Perhaps a better way forward is to not follow the abstract-level interface at all, if the requirements are too different.

Increase consistency with reads, i.e. writes are immediately visible for preceding read calls. Without having to add a lot of complicated write barriers and read queuing in the binding.

You have a fundamentally different idea of what the order of operations should be. Which is fine, but it's not level. Level promises consistency in a simple way: want to know that a write finished? Await it. Subsequent reads are then guaranteed to include that written data.

Avoid problems with writes being applied out of order due to thread pool scheduling.

Same applies here: userland code can decide (to care about) the order, by awaiting.