search

ssbc / jitdb

A database on top of a log with automatic index generation and maintenance
50 stars 7 forks source link

New core index: deleted.index #237

Open staltz opened 1 year ago

staltz commented 1 year ago

:bulb: Idea

When we're streaming the log in jitdb, we can detect deleted records, but we don't really do anything about it, we just skip to the next records. What we could do is update a bitvector deleted.index to indicate that such record is deleted. Then, this can alleviate other code downstream, such that we don't have to get the record from the log, verify that it's a deleted record (which by the way uses the is-buffer-zero module), and then drop it from the query results. We can just do a bitvector AND with deleted.index.

Since deletes can happens in random order, we will also have to listen to a stream of deleted offsets from AAOL. Such API doesn't currently exist in AAOL but could be somewhat easily built, maybe as an enhancement of AAOL.onDeletesFlushed().

cc @arj03

arj03 commented 1 year ago

True, this could be a good speedup, especially if you have many deleted messages. Could just be a pre-filter, we still have the check in get which is good to keep there in case things are not totally in sync.