search

ssbc / ssb-db2

A new database for secure-scuttlebutt
47 stars 8 forks source link

do not allow concurrent updateIndexes() calls #362

Closed staltz closed 2 years ago

staltz commented 2 years ago

Context

I was testing Manyverse and noticed with DEBUG=ssb:db2 that there were really two different updateIndexes() calls running at startup (in normal use case circumstances).

Problem

ReadyGate.onReady adds the cb to an array, even if that cb was already added to the array. So two different calls of onReady are not idempotent, causing the cb to be run twice when the ReadyGate is ready.

Solution

Use a Set of functions instead of an Array of functions, inside ReadyGate.

I also added a throw in updateIndexes() in case there is another updateIndexes() ongoing. The purpose of this was to fail the tests, but strangely, the process exit code is green, even though the test suite "failed". Weird, and searching for unhandledRejection in tape issues left me confused.

arj03 commented 2 years ago

The set change looks great :)

github-actions[bot] commented 2 years ago

Benchmark results

Part Duration
add 1000 elements 497ms
add 1000 private box1 elements 1212ms
unbox 1000 private box1 elements first run 127ms
unbox 1000 private box1 elements second run 97ms
add 1000 private box1 elements 1197ms
query 1000 elements first run 36ms
query 1000 elements second run 12ms
add 1000 private box2 elements 597ms
unbox 1000 private box2 elements first run 354ms
unbox 1000 private box2 elements second run 351ms
Migrate (+db1) 10956ms
Migrate (alone) 3631ms
Migrate (+db1 +db2) 8198ms
Migrate (+db2) 5951ms
Migrate continuation (+db2) 843ms
Memory usage without indexes 737.21 MB = 37.13 MB + etc
Initial indexing 640ms
Initial indexing maxCpu=86 3956ms
Initial indexing compat 625ms
Two indexes updating concurrently 850ms
key one initial 51ms
key two 0ms
key one again 1ms
reboot and key one again 47ms
latest root posts 540ms
latest posts 5ms
votes one initial 529ms
votes again 0ms
hasRoot 341ms
hasRoot again 1ms
author one posts 307ms
author two posts 16ms
dedicated author one posts 306ms
dedicated author one posts again 0ms
Maximum memory usage 1010.69 MB = 63.67 MB + etc
Indexes folder size 9.97mb
github-actions[bot] commented 2 years ago

Benchmark results

Part Duration
add 1000 elements 508ms
add 1000 private box1 elements 1471ms
unbox 1000 private box1 elements first run 176ms
unbox 1000 private box1 elements second run 131ms
add 1000 private box1 elements 1429ms
query 1000 elements first run 41ms
query 1000 elements second run 14ms
add 1000 private box2 elements 812ms
unbox 1000 private box2 elements first run 496ms
unbox 1000 private box2 elements second run 481ms
Migrate (+db1) 15859ms
Migrate (alone) 5289ms
Migrate (+db1 +db2) 11631ms
Migrate (+db2) 7962ms
Migrate continuation (+db2) 1251ms
Memory usage without indexes 739.24 MB = 30.44 MB + etc
Initial indexing 910ms
Initial indexing maxCpu=86 4983ms
Initial indexing compat 1015ms
Two indexes updating concurrently 1283ms
key one initial 62ms
key two 2ms
key one again 1ms
reboot and key one again 67ms
latest root posts 780ms
latest posts 22ms
votes one initial 621ms
votes again 1ms
hasRoot 465ms
hasRoot again 0ms
author one posts 412ms
author two posts 17ms
dedicated author one posts 367ms
dedicated author one posts again 0ms
Maximum memory usage 942.92 MB = 58.56 MB + etc
Indexes folder size 9.97mb