search

ssbc / jitdb

A database on top of a log with automatic index generation and maintenance
50 stars 7 forks source link

prepare() updates status when starting #203

Closed staltz closed 2 years ago

staltz commented 2 years ago

Context

https://github.com/ssb-ngi-pointer/jitdb/pull/202

Problem

In order to get a predictable progress bar, the status has to be updated at the beginning of jitdb.prepare(), but we were only updating the status periodically (every 1000 records) during the building of the index.

Solution

Call status.batchUpdate() at the beginning of executeOperation.

1st :x: 2nd :heavy_check_mark:

github-actions[bot] commented 2 years ago

Benchmark results

Part Speed Heap Change Samples
Count 1 big index (3rd run) 0.55ms ± 0.27ms 2.91 kB ± 11.39 kB 51
Create an index twice concurrently 747.87ms ± 6.63ms -60.47 kB ± 79.2 kB 73
Load core indexes 1.31ms ± 0.02ms 93.22 B ± 225.79 B 7082
Load two indexes concurrently 724.56ms ± 4.72ms 60.62 kB ± 142.06 kB 17
Paginate 10 results 25.71ms ± 0.95ms 6.47 kB ± 11.49 kB 27
Paginate 20000 msgs with pageSize=5 19902.01ms ± 469.29ms 3.15 MB ± 2.68 MB 5
Paginate 20000 msgs with pageSize=500 581.07ms ± 5.65ms -17.97 kB ± 1144.63 kB 18
Query 1 big index (1st run) 961.57ms ± 16.32ms 41.35 kB ± 101.22 kB 56
Query 1 big index (2nd run) 270.56ms ± 3.21ms -20.49 kB ± 32.84 kB 49
Query 3 indexes (1st run) 1079.36ms ± 12.63ms -53.99 kB ± 138.28 kB 50
Query 3 indexes (2nd run) 221.81ms ± 1.16ms 126.72 kB ± 169.74 kB 53
Query a prefix map (1st run) 238.15ms ± 1.9ms -38.2 kB ± 102.01 kB 24
Query a prefix map (2nd run) 14.93ms ± 0.77ms -9.21 kB ± 10.62 kB 24
github-actions[bot] commented 2 years ago

Benchmark results

Part Speed Heap Change Samples
Count 1 big index (3rd run) 0.38ms ± 0.03ms 7.73 kB ± 24.06 kB 41
Create an index twice concurrently 735ms ± 9.56ms 9.21 kB ± 72.26 kB 74
Load core indexes 1.41ms ± 0.02ms 95.95 B ± 211.99 B 6820
Load two indexes concurrently 483.13ms ± 7.14ms 33.87 kB ± 123.52 kB 21
Paginate 10 results 22.62ms ± 1.03ms 1.38 kB ± 11.5 kB 27
Paginate 20000 msgs with pageSize=5 7410.06ms ± 76.45ms -195.68 kB ± 5767.67 kB 5
Paginate 20000 msgs with pageSize=500 552.33ms ± 25.25ms -111.05 kB ± 426.17 kB 20
Query 1 big index (1st run) 796.08ms ± 14.69ms 8 kB ± 74.59 kB 69
Query 1 big index (2nd run) 310.1ms ± 3.47ms -8.28 kB ± 44.49 kB 43
Query 3 indexes (1st run) 848.7ms ± 9.36ms 15.75 kB ± 130.67 kB 64
Query 3 indexes (2nd run) 229.78ms ± 1.24ms 36.23 kB ± 177.52 kB 52
Query a prefix map (1st run) 241.55ms ± 2.68ms -35.67 kB ± 411.48 kB 24
Query a prefix map (2nd run) 13.55ms ± 0.56ms -5.56 kB ± 21.43 kB 24