search

ssbc / ssb-db2

A new database for secure-scuttlebutt
47 stars 8 forks source link

improve performance of del() by 20x #336

Closed staltz closed 2 years ago

staltz commented 2 years ago

Uses jitdb.lookup() (see https://github.com/ssb-ngi-pointer/jitdb/issues/215) to improve the performance of del() and getMsg().

See the numbers for del():

Before

Duration
Publish 1000 messages 359.185ms
Delete 500 messages 1039.531ms

After

Duration
Publish 1000 messages 350.658ms
Delete 500 messages 36.623ms
Wait for deletes to flush 264.364ms

Note that the flushing of deletes (264ms) is probably dominated by the duration of the debounce period which is 250ms.

github-actions[bot] commented 2 years ago

Benchmark results

Part Duration
add 1000 elements 502ms
add 1000 private box1 elements 1477ms
unbox 1000 private box1 elements first run 167ms
unbox 1000 private box1 elements second run 129ms
add 1000 private box1 elements 1433ms
query 1000 elements first run 37ms
query 1000 elements second run 17ms
add 1000 private box2 elements 795ms
unbox 1000 private box2 elements first run 413ms
unbox 1000 private box2 elements second run 398ms
Migrate (+db1) 14794ms
Migrate (alone) 5069ms
Migrate (+db1 +db2) 10702ms
Migrate (+db2) 7895ms
Migrate continuation (+db2) 1398ms
Memory usage without indexes 719.79 MB = 30.53 MB + etc
Initial indexing 766ms
Initial indexing maxCpu=86 3792ms
Initial indexing compat 1030ms
Two indexes updating concurrently 928ms
key one initial 52ms
key two 3ms
key one again 3ms
reboot and key one again 53ms
latest root posts 716ms
latest posts 16ms
votes one initial 608ms
votes again 1ms
hasRoot 444ms
hasRoot again 1ms
author one posts 422ms
author two posts 20ms
dedicated author one posts 395ms
dedicated author one posts again 0ms
Maximum memory usage 857.24 MB = 85.54 MB + etc
Indexes folder size 9.97mb
staltz commented 2 years ago

I could be interesting to add a console.log to this getMsg and fire up Manyverse to see how many times that method is called within say the first 1 minute of "normal" usage.

This was easy to check! getMsg was called 53 times within the first 10 seconds after startup. After that, it called getMsg more only if I scrolled down (because it loaded more from ssb-threads), but not too much, about +3 calls to getMsg for every new thread loaded on the screen.

arj03 commented 2 years ago

Great, so it should also help a tiny bit overall when using ssb-threads. Good to hear! :)

github-actions[bot] commented 2 years ago

Benchmark results

Part Duration
add 1000 elements 513ms
add 1000 private box1 elements 1146ms
unbox 1000 private box1 elements first run 122ms
unbox 1000 private box1 elements second run 88ms
add 1000 private box1 elements 1139ms
query 1000 elements first run 40ms
query 1000 elements second run 15ms
add 1000 private box2 elements 631ms
unbox 1000 private box2 elements first run 330ms
unbox 1000 private box2 elements second run 316ms
Migrate (+db1) 10974ms
Migrate (alone) 3679ms
Migrate (+db1 +db2) 7834ms
Migrate (+db2) 5795ms
Migrate continuation (+db2) 924ms
Memory usage without indexes 724.43 MB = 30.85 MB + etc
Initial indexing 620ms
Initial indexing maxCpu=86 3958ms
Initial indexing compat 866ms
Two indexes updating concurrently 790ms
key one initial 47ms
key two 1ms
key one again 1ms
reboot and key one again 44ms
latest root posts 576ms
latest posts 8ms
votes one initial 463ms
votes again 0ms
hasRoot 323ms
hasRoot again 1ms
author one posts 301ms
author two posts 14ms
dedicated author one posts 278ms
dedicated author one posts again 1ms
Maximum memory usage 806.05 MB = 52.32 MB + etc
Indexes folder size 9.97mb