Closed staltz closed 2 years ago
Oh, benchmark found a problem :)
I didn't see that stop was an exposed function
Part | Duration |
---|---|
add 1000 elements | 500ms |
add 1000 private box1 elements | 1282ms |
unbox 1000 private box1 elements first run | 139ms |
unbox 1000 private box1 elements second run | 101ms |
add 1000 private box1 elements | 1260ms |
query 1000 elements first run | 35ms |
query 1000 elements second run | 17ms |
add 1000 private box2 elements | 696ms |
unbox 1000 private box2 elements first run | 405ms |
unbox 1000 private box2 elements second run | 374ms |
Migrate (+db1) | 12785ms |
Migrate (alone) | 4705ms |
Migrate (+db1 +db2) | 10444ms |
Migrate (+db2) | 7777ms |
Migrate continuation (+db2) | 1252ms |
Memory usage without indexes | 1089.86 MB = 38.16 MB + etc |
Initial indexing | 854ms |
Initial indexing maxCpu=86 | 5713ms |
Initial indexing compat | 1016ms |
Two indexes updating concurrently | 1122ms |
key one initial | 53ms |
key two | 2ms |
key one again | 2ms |
reboot and key one again | 54ms |
latest root posts | 726ms |
latest posts | 5ms |
votes one initial | 619ms |
votes again | 1ms |
hasRoot | 438ms |
hasRoot again | 1ms |
author one posts | 376ms |
author two posts | 20ms |
dedicated author one posts | 383ms |
dedicated author one posts again | 1ms |
Maximum memory usage | 1500.36 MB = 66.18 MB + etc |
Indexes folder size | 9.97mb |
@arj03 Well spotted. I was lazy to change stop() to async (and make a breaking change version) so I went with a simpler alternative instead, does almost the same.
So you don't need all what stop() does at this place? Like drainAborter.abort etc.? I'm not super familiar with this code.
Well, things were working before without calling stop(), but I guess I can add stop()
to the close hook.
It's a shame that flumelog doesn't have a close()
, otherwise I'd call that too, on stop().
Part | Duration |
---|---|
add 1000 elements | 507ms |
add 1000 private box1 elements | 1213ms |
unbox 1000 private box1 elements first run | 119ms |
unbox 1000 private box1 elements second run | 102ms |
add 1000 private box1 elements | 1202ms |
query 1000 elements first run | 32ms |
query 1000 elements second run | 12ms |
add 1000 private box2 elements | 597ms |
unbox 1000 private box2 elements first run | 353ms |
unbox 1000 private box2 elements second run | 358ms |
Migrate (+db1) | 10602ms |
Migrate (alone) | 3609ms |
Migrate (+db1 +db2) | 8427ms |
Migrate (+db2) | 6082ms |
Migrate continuation (+db2) | 830ms |
Memory usage without indexes | 734.09 MB = 30.37 MB + etc |
Initial indexing | 655ms |
Initial indexing maxCpu=86 | 4405ms |
Initial indexing compat | 809ms |
Two indexes updating concurrently | 859ms |
key one initial | 53ms |
key two | 0ms |
key one again | 0ms |
reboot and key one again | 45ms |
latest root posts | 566ms |
latest posts | 9ms |
votes one initial | 519ms |
votes again | 1ms |
hasRoot | 344ms |
hasRoot again | 0ms |
author one posts | 310ms |
author two posts | 17ms |
dedicated author one posts | 315ms |
dedicated author one posts again | 0ms |
Maximum memory usage | 1006.42 MB = 60.16 MB + etc |
Indexes folder size | 9.97mb |
Part | Duration |
---|---|
add 1000 elements | 1706ms |
add 1000 private box1 elements | 930ms |
unbox 1000 private box1 elements first run | 101ms |
unbox 1000 private box1 elements second run | 74ms |
add 1000 private box1 elements | 917ms |
query 1000 elements first run | 31ms |
query 1000 elements second run | 10ms |
add 1000 private box2 elements | 494ms |
unbox 1000 private box2 elements first run | 252ms |
unbox 1000 private box2 elements second run | 250ms |
Migrate (+db1) | 18769ms |
Migrate (alone) | 10997ms |
Migrate (+db1 +db2) | 12948ms |
Migrate (+db2) | 13203ms |
Migrate continuation (+db2) | 764ms |
Memory usage without indexes | 700.69 MB = 34.63 MB + etc |
Initial indexing | 575ms |
Initial indexing maxCpu=86 | 3849ms |
Initial indexing compat | 670ms |
Two indexes updating concurrently | 839ms |
key one initial | 84ms |
key two | 1ms |
key one again | 1ms |
reboot and key one again | 82ms |
latest root posts | 481ms |
latest posts | 5ms |
votes one initial | 430ms |
votes again | 1ms |
hasRoot | 265ms |
hasRoot again | 1ms |
author one posts | 266ms |
author two posts | 14ms |
dedicated author one posts | 223ms |
dedicated author one posts again | 0ms |
Maximum memory usage | 989.54 MB = 62.88 MB + etc |
Indexes folder size | 9.97mb |
Context
I was testing Manyverse, and noticed that right after it migrates
~/.ssb
to ssb-db2 (usingssb-db2/migrate
without the rest of ssb-db2), indexing would not happen at all, the app seemed totally blank without content. On a second run of the app, indexing happens and content appears.Problem
When using the migrate plugin alone, without ssb-db2, then the
sbot.close
hook does not runlog.close
(which should make sure all writes are done). This meant thatsbot.close
callbacks were called before AAOL was reading writing to disk.Solution
ssb-db2/db.js already handles this case correctly, so we only need to patch the case where we are running
.use(require('ssb-db2/migrate'))
without.use(require('ssb-db'))
, such that the close hook callslog.close
.1st :x: 2nd :heavy_check_mark: