ssbc / jitdb

A database on top of a log with automatic index generation and maintenance
50 stars 7 forks source link

add APIs necessary for log compaction algorithms #217

Closed staltz closed 2 years ago

staltz commented 2 years ago

Context: https://github.com/ssb-ngi-pointer/jitdb/issues/199

Problem 1: we need to know when is indexing ongoing and/or when is querying ongoing, because we don't want compact to happen in parallel with those.

Problem 2: if compaction is ongoing, we can't allow JITDB queries to start.

Problem 3: reindex has to rebuild also the core indexes.

Solution:

These changes are already used and tested in https://github.com/ssb-ngi-pointer/ssb-db2/pull/339

github-actions[bot] commented 2 years ago

Benchmark results

Part Speed Heap Change Samples
Count 1 big index (3rd run) 0.38ms ± 0.03ms -1.8 kB ± 10.57 kB 59
Create an index twice concurrently 500.15ms ± 2.67ms 18.21 kB ± 25.39 kB 109
Load core indexes 0.8ms ± 0.01ms 92.7 B ± 181.62 B 9685
Load two indexes concurrently 407.67ms ± 2.56ms -83.75 kB ± 106.33 kB 24
Paginate 10 results 26.35ms ± 1.64ms 11.51 kB ± 16.76 kB 29
Paginate 20000 msgs with pageSize=5 6103.73ms ± 86.69ms 145.33 kB ± 973.87 kB 5
Paginate 20000 msgs with pageSize=500 480.78ms ± 2.46ms 231.13 kB ± 415.86 kB 23
Query 1 big index (1st run) 742.65ms ± 11.93ms -2.83 kB ± 38.49 kB 74
Query 1 big index (2nd run) 251.61ms ± 0.62ms 1.71 kB ± 10.18 kB 59
Query 3 indexes (1st run) 1068.12ms ± 14.93ms -52.37 kB ± 150.24 kB 50
Query 3 indexes (2nd run) 218.64ms ± 1.08ms 21.41 kB ± 24.41 kB 62
Query a prefix map (1st run) 225.85ms ± 1.94ms 28.97 kB ± 534.77 kB 26
Query a prefix map (2nd run) 14ms ± 0.7ms -12.99 kB ± 45.61 kB 26