Closed VladRodionov closed 2 months ago
We can avoid rehashing index as well before saving. This will give us additional performance boost.
First optimization: increased performance of reading saved memory index by introducing BufferedInputStream with large buffer size (16MB) This gives significant boost in read I/O. Another optimization is parallel saving loading engine data and memory index.
Further optimizations seems won't give any significant reduction in save/load cache operation time. We are now close to 4GB/s on load and 2GB/s on save on Mac Studio M1
Removed finishRehashing()
call in MemoryIndex::save()
method - it improved save operation performance.
For in-memory caches we can write segments to disk asynchronously when segment is sealed. Then on shutdown we will need to save only index and some other meta , this should be much faster. On start up, read data segments in parallel.