search

ssbc / ssb-db2

A new database for secure-scuttlebutt
47 stars 8 forks source link

Test and fix post compat #426

Closed Powersource closed 10 months ago

Powersource commented 10 months ago

For https://github.com/ssbc/ssb-tribes/pull/105

Trying to remove db1 completely from ssb-tribes, apparently some test runs replicate and that depends on sbot.post. But the compat plugin was only providing sbot.db.post. I also added a test.

Powersource commented 10 months ago

@staltz review if you have time :) otherwise i could pass to mix probably

Powersource commented 10 months ago

damn that's not something i expected to fail

migrate keeps new log synced with ssb-db being updated
2023-11-02T19:33:14.129Z ssb:db2:private encrypted or decrypted indexes seem to be empty
2023-11-02T19:33:14.133Z ssb:db2 lowest offset for all indexes is -1
2023-11-02T19:33:14.133Z ssb:db2 updateIndexes() scan time: 0ms
2023-11-02T19:33:14.133Z ssb:db2 updateIndexes() live streaming
    ✔ migration done
    ✔ 1st query suceeded
2023-11-02T19:33:14.136Z ssb:db2 getAllLatest is done setting up initial validate state
2023-11-02T19:33:14.136Z ssb:db2 getAllLatest is done setting up initial validate state
    ✔ 10 msgs
    ✔ publish suceeded
    ✔ msg posted
    ✔ 2nd query suceeded
Uncaught Exception at  uncaughtException : TypeError: Cannot read properties of undefined (reading 'value')
    at /home/runner/work/ssb-db2/ssb-db2/test/migration.js:144:34
    at /home/runner/work/ssb-db2/ssb-db2/node_modules/jitdb/index.js:1417:13
    at CollectStream._cb (/home/runner/work/ssb-db2/ssb-db2/node_modules/jitdb/index.js:1247:11)
    at CollectStream.end (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/sinks/collect.js:16:15)
    at FilterStream.end (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/throughs/through.js:23:22)
    at AsyncMapStream.end (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/throughs/async-map.js:28:20)
    at ValueStream.resume (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/sources/values.js:19:51)
    at AsyncMapStream.resume (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/throughs/through.js:17:19)
    at FilterStream.resume (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/throughs/through.js:17:19)
    at FilterStream.pipe (/home/runner/work/ssb-db2/ssb-db2/node_modules/push-stream/pipe.js:18:26)
    ✖ 418) 11 msgs
      Expected 11 but got 10
      At: <anonymous> (/test/migration.js:143:19)
Powersource commented 10 months ago

so uugh code seems to be supposed to allow importing db1 and db2 at the same time? which breaks with this.

trying out removing post compat from the default imports

Powersource commented 10 months ago

yay works now

Powersource commented 10 months ago

i guess you can argue this is a major version since it changes the default imports but eeh no one uses those anyway

Powersource commented 10 months ago

@staltz done for real now

github-actions[bot] commented 10 months ago

Benchmark results

Part Duration
Create 5000 new messages 328.35ms
Validate 5000 messages 427.60ms
Native to db format 5000 messages 104.93ms
Db to native format 5000 messages 88.56ms
Add 1000 elements 361.98ms
Add 1000 box1 msgs 913.16ms
Unbox 1000 box1 msgs first run 177.58ms
Unbox 1000 box1 msgs second run 87.96ms
Add 1000 box1 msgs 859.41ms
Query 1000 msgs first run 41.25ms
Query 1000 msgs second run 10.58ms
Add 1000 box2 msgs 1317.45ms
Unbox 1000 box2 msgs first run 369.27ms
Unbox 1000 box2 msgs second run 174.19ms
Migrate (+db1) 10860.40ms
Migrate (alone) 3703.97ms
Migrate (+db1 +db2) 7600.90ms
Migrate (+db2) 5431.51ms
Migrate continuation (+db2) 1043.56ms
Memory usage without indexes 815.01 MB = 43.38 MB + etc
Initial indexing 1467.35ms
Initial indexing maxcpu=86 4830.18ms
Initial indexing compat 636.66ms
Two indexes updating concurrently 2517.12ms
Key one initial 86.17ms
Key two 1.26ms
Key one again 0.98ms
Reboot and key one again 79.09ms
Latest root posts 1629.58ms
Latest posts 10.25ms
Votes one initial 1290.67ms
Votes again 0.40ms
HasRoot 1169.69ms
HasRoot again 0.38ms
Author one posts 1251.42ms
Author two posts 26.00ms
Dedicated author one posts 1251.48ms
Dedicated author one posts again 0.92ms
DeleteFeed 5337.18ms
Maximum memory usage 861.89 MB = 59.41 MB + etc
Indexes folder size 10.01mb
Powersource commented 10 months ago

v8.0.0

github-actions[bot] commented 10 months ago

Benchmark results

Part Duration
Create 5000 new messages 358.80ms
Validate 5000 messages 456.62ms
Native to db format 5000 messages 92.58ms
Db to native format 5000 messages 90.28ms
Add 1000 elements 366.86ms
Add 1000 box1 msgs 980.34ms
Unbox 1000 box1 msgs first run 186.39ms
Unbox 1000 box1 msgs second run 89.45ms
Add 1000 box1 msgs 934.41ms
Query 1000 msgs first run 36.88ms
Query 1000 msgs second run 11.18ms
Add 1000 box2 msgs 1301.77ms
Unbox 1000 box2 msgs first run 312.52ms
Unbox 1000 box2 msgs second run 156.42ms
Migrate (+db1) 10587.18ms
Migrate (alone) 3374.11ms
Migrate (+db1 +db2) 7440.20ms
Migrate (+db2) 5249.44ms
Migrate continuation (+db2) 941.62ms
Memory usage without indexes 763.99 MB = 39.08 MB + etc
Initial indexing 1403.12ms
Initial indexing maxcpu=86 4740.16ms
Initial indexing compat 538.03ms
Two indexes updating concurrently 2459.00ms
Key one initial 65.35ms
Key two 1.10ms
Key one again 0.92ms
Reboot and key one again 79.46ms
Latest root posts 1567.41ms
Latest posts 25.31ms
Votes one initial 1254.46ms
Votes again 0.36ms
HasRoot 1198.69ms
HasRoot again 0.30ms
Author one posts 1266.28ms
Author two posts 29.32ms
Dedicated author one posts 1275.11ms
Dedicated author one posts again 0.49ms
DeleteFeed 5000.25ms
Maximum memory usage 910.52 MB = 60.65 MB + etc
Indexes folder size 10.01mb