search

guycipher / k4

High-performance open-source, durable, transactional embedded storage engine designed for low-latency, and optimized read and write efficiency.
https://pkg.go.dev/github.com/guycipher/k4/v2
BSD 3-Clause "New" or "Revised" License
250 stars 6 forks source link

Background flushing and compaction #2

Closed guycipher closed 1 month ago

guycipher commented 1 month ago

Currently K4 will block on flushing and compaction, though not long depending on memtable threshold, size of sstables, etc. It would be ideal to run these operations in the background in a GO routine. When flushing memtable we can make a copy of maxed memtable, clear the memtable so new operations can come in and continue with flush that way. In regards to compaction we can do something similar. This would narrow down the blocking on flushing and compaction operations.

guycipher commented 1 month ago

Working on this at the moment.

guycipher commented 1 month ago

Working rather good locally. I'm working through a few tests as the background operations need to handled gracefully on shutdown, making sure whatever flushes need to occur are escalated.

guycipher commented 1 month ago

I seem to be getting false positives from the bloom filter on sstables which is throwing me off a bit. I am investigating this and see what optimizations I can do here.

guycipher commented 1 month ago

Ah no never mind, you know what, tracing back some commits the flush test broke after compression implementation, I am further investigating.

guycipher commented 1 month ago

Completed in v1.5.0b