search

ssbc / ssb-db2

A new database for secure-scuttlebutt
47 stars 8 forks source link

fix migrate: close newLog when there is no ssb-db2 installed #361

Closed staltz closed 2 years ago

staltz commented 2 years ago

Context

I was testing Manyverse, and noticed that right after it migrates ~/.ssb to ssb-db2 (using ssb-db2/migrate without the rest of ssb-db2), indexing would not happen at all, the app seemed totally blank without content. On a second run of the app, indexing happens and content appears.

Problem

When using the migrate plugin alone, without ssb-db2, then the sbot.close hook does not run log.close (which should make sure all writes are done). This meant that sbot.close callbacks were called before AAOL was reading writing to disk.

Solution

ssb-db2/db.js already handles this case correctly, so we only need to patch the case where we are running .use(require('ssb-db2/migrate')) without .use(require('ssb-db')), such that the close hook calls log.close.

1st :x: 2nd :heavy_check_mark:

arj03 commented 2 years ago

Oh, benchmark found a problem :)

arj03 commented 2 years ago

This should probably also be async now?

arj03 commented 2 years ago

I didn't see that stop was an exposed function

github-actions[bot] commented 2 years ago

Benchmark results

Part Duration
add 1000 elements 500ms
add 1000 private box1 elements 1282ms
unbox 1000 private box1 elements first run 139ms
unbox 1000 private box1 elements second run 101ms
add 1000 private box1 elements 1260ms
query 1000 elements first run 35ms
query 1000 elements second run 17ms
add 1000 private box2 elements 696ms
unbox 1000 private box2 elements first run 405ms
unbox 1000 private box2 elements second run 374ms
Migrate (+db1) 12785ms
Migrate (alone) 4705ms
Migrate (+db1 +db2) 10444ms
Migrate (+db2) 7777ms
Migrate continuation (+db2) 1252ms
Memory usage without indexes 1089.86 MB = 38.16 MB + etc
Initial indexing 854ms
Initial indexing maxCpu=86 5713ms
Initial indexing compat 1016ms
Two indexes updating concurrently 1122ms
key one initial 53ms
key two 2ms
key one again 2ms
reboot and key one again 54ms
latest root posts 726ms
latest posts 5ms
votes one initial 619ms
votes again 1ms
hasRoot 438ms
hasRoot again 1ms
author one posts 376ms
author two posts 20ms
dedicated author one posts 383ms
dedicated author one posts again 1ms
Maximum memory usage 1500.36 MB = 66.18 MB + etc
Indexes folder size 9.97mb
staltz commented 2 years ago

@arj03 Well spotted. I was lazy to change stop() to async (and make a breaking change version) so I went with a simpler alternative instead, does almost the same.

arj03 commented 2 years ago

So you don't need all what stop() does at this place? Like drainAborter.abort etc.? I'm not super familiar with this code.

staltz commented 2 years ago

Well, things were working before without calling stop(), but I guess I can add stop() to the close hook.

It's a shame that flumelog doesn't have a close(), otherwise I'd call that too, on stop().

github-actions[bot] commented 2 years ago

Benchmark results

Part Duration
add 1000 elements 507ms
add 1000 private box1 elements 1213ms
unbox 1000 private box1 elements first run 119ms
unbox 1000 private box1 elements second run 102ms
add 1000 private box1 elements 1202ms
query 1000 elements first run 32ms
query 1000 elements second run 12ms
add 1000 private box2 elements 597ms
unbox 1000 private box2 elements first run 353ms
unbox 1000 private box2 elements second run 358ms
Migrate (+db1) 10602ms
Migrate (alone) 3609ms
Migrate (+db1 +db2) 8427ms
Migrate (+db2) 6082ms
Migrate continuation (+db2) 830ms
Memory usage without indexes 734.09 MB = 30.37 MB + etc
Initial indexing 655ms
Initial indexing maxCpu=86 4405ms
Initial indexing compat 809ms
Two indexes updating concurrently 859ms
key one initial 53ms
key two 0ms
key one again 0ms
reboot and key one again 45ms
latest root posts 566ms
latest posts 9ms
votes one initial 519ms
votes again 1ms
hasRoot 344ms
hasRoot again 0ms
author one posts 310ms
author two posts 17ms
dedicated author one posts 315ms
dedicated author one posts again 0ms
Maximum memory usage 1006.42 MB = 60.16 MB + etc
Indexes folder size 9.97mb
github-actions[bot] commented 2 years ago

Benchmark results

Part Duration
add 1000 elements 1706ms
add 1000 private box1 elements 930ms
unbox 1000 private box1 elements first run 101ms
unbox 1000 private box1 elements second run 74ms
add 1000 private box1 elements 917ms
query 1000 elements first run 31ms
query 1000 elements second run 10ms
add 1000 private box2 elements 494ms
unbox 1000 private box2 elements first run 252ms
unbox 1000 private box2 elements second run 250ms
Migrate (+db1) 18769ms
Migrate (alone) 10997ms
Migrate (+db1 +db2) 12948ms
Migrate (+db2) 13203ms
Migrate continuation (+db2) 764ms
Memory usage without indexes 700.69 MB = 34.63 MB + etc
Initial indexing 575ms
Initial indexing maxCpu=86 3849ms
Initial indexing compat 670ms
Two indexes updating concurrently 839ms
key one initial 84ms
key two 1ms
key one again 1ms
reboot and key one again 82ms
latest root posts 481ms
latest posts 5ms
votes one initial 430ms
votes again 1ms
hasRoot 265ms
hasRoot again 1ms
author one posts 266ms
author two posts 14ms
dedicated author one posts 223ms
dedicated author one posts again 0ms
Maximum memory usage 989.54 MB = 62.88 MB + etc
Indexes folder size 9.97mb