staltz
commented
2 years ago I broke tests elsewhere, but I'll check that tomorrow. @arj03 this is ready for a first-pass review (I have checked that the solution passes the test I added)
Closed staltz closed 2 years ago
staltz
commented
2 years ago I broke tests elsewhere, but I'll check that tomorrow. @arj03 this is ready for a first-pass review (I have checked that the solution passes the test I added)
staltz
commented
2 years ago @arj03 Okay, I think all tests pass now. What I did in the https://github.com/ssbc/ssb-db2/pull/386/commits/d7bc8644a13ce99a202d5db283f6af117dd6f778 commit was to allow jitdb log.streams to decrypt messages (and create the msg.meta), but not update the private index latestOffset. That's the sole responsibility of the core.js log.stream that updates leveldb indexes.
github-actions[bot]
commented
2 years ago | Part | Duration |
|---|---|
| Create 5000 new messages | 442.39ms |
| Validate 5000 messages | 532.82ms |
| Native to db format 5000 messages | 174.20ms |
| Db to native format 5000 messages | 105.36ms |
| Add 1000 elements | 521.60ms |
| Add 1000 box1 msgs | 1158.90ms |
| Unbox 1000 box1 msgs first run | 275.24ms |
| Unbox 1000 box1 msgs second run | 135.22ms |
| Add 1000 box1 msgs | 1154.96ms |
| Query 1000 msgs first run | 49.63ms |
| Query 1000 msgs second run | 28.32ms |
| Add 1000 box2 msgs | 1584.13ms |
| Unbox 1000 box2 msgs first run | 379.14ms |
| Unbox 1000 box2 msgs second run | 190.25ms |
| Migrate (+db1) | 15364.42ms |
| Migrate (alone) | 5190.87ms |
| Migrate (+db1 +db2) | 11003.63ms |
| Migrate (+db2) | 7717.33ms |
| Migrate continuation (+db2) | 1210.87ms |
| Memory usage without indexes | 743.41 MB = 38.40 MB + etc |
| Initial indexing | 1998.42ms |
| Initial indexing maxcpu=86 | 5860.72ms |
| Initial indexing compat | 1039.87ms |
| Two indexes updating concurrently | 3705.31ms |
| Key one initial | 61.41ms |
| Key two | 1.58ms |
| Key one again | 1.28ms |
| Reboot and key one again | 61.68ms |
| Latest root posts | 2276.09ms |
| Latest posts | 13.97ms |
| Votes one initial | 1902.46ms |
| Votes again | 0.78ms |
| HasRoot | 1696.16ms |
| HasRoot again | 0.35ms |
| Author one posts | 1655.69ms |
| Author two posts | 38.04ms |
| Dedicated author one posts | 1749.02ms |
| Dedicated author one posts again | 0.72ms |
| DeleteFeed | 3764.07ms |
| Maximum memory usage | 1008.03 MB = 63.67 MB + etc |
| Indexes folder size | 10.12mb |
github-actions[bot]
commented
2 years ago | Part | Duration |
|---|---|
| Create 5000 new messages | 363.46ms |
| Validate 5000 messages | 482.49ms |
| Native to db format 5000 messages | 111.12ms |
| Db to native format 5000 messages | 80.80ms |
| Add 1000 elements | 502.24ms |
| Add 1000 box1 msgs | 1006.34ms |
| Unbox 1000 box1 msgs first run | 232.13ms |
| Unbox 1000 box1 msgs second run | 111.86ms |
| Add 1000 box1 msgs | 968.66ms |
| Query 1000 msgs first run | 45.85ms |
| Query 1000 msgs second run | 12.31ms |
| Add 1000 box2 msgs | 1286.27ms |
| Unbox 1000 box2 msgs first run | 304.32ms |
| Unbox 1000 box2 msgs second run | 144.10ms |
| Migrate (+db1) | 11026.73ms |
| Migrate (alone) | 3803.05ms |
| Migrate (+db1 +db2) | 8267.18ms |
| Migrate (+db2) | 5804.84ms |
| Migrate continuation (+db2) | 876.67ms |
| Memory usage without indexes | 763.89 MB = 37.95 MB + etc |
| Initial indexing | 1600.45ms |
| Initial indexing maxcpu=86 | 5090.83ms |
| Initial indexing compat | 720.41ms |
| Two indexes updating concurrently | 2827.67ms |
| Key one initial | 46.78ms |
| Key two | 2.15ms |
| Key one again | 1.78ms |
| Reboot and key one again | 45.88ms |
| Latest root posts | 1721.03ms |
| Latest posts | 8.77ms |
| Votes one initial | 1483.19ms |
| Votes again | 0.48ms |
| HasRoot | 1289.47ms |
| HasRoot again | 0.29ms |
| Author one posts | 1390.27ms |
| Author two posts | 23.54ms |
| Dedicated author one posts | 1439.54ms |
| Dedicated author one posts again | 0.84ms |
| DeleteFeed | 2746.57ms |
| Maximum memory usage | 994.79 MB = 59.75 MB + etc |
| Indexes folder size | 10.01mb |
Context
https://github.com/ssbc/ssb-replication-scheduler/issues/3
Problem
In privateIndex we use
streaming === trueto update the latestOffset, but this is troublesome if there is some live stream which takes the latest record in the log, because it'll update privateIndex's latestOffset and then when reindexing all previous records, we won't attempt to decrypt them because they aren't in the decrypted list and neither does therecOffset > latestOffsetpass.Solution
We should really only have one log.stream that is responsible for updating the private index. I elect the "leveldb log.stream" to be the only stream that can do that, thus we pass a
opts.shouldDecrypt = trueonly for the core log.stream.At the same time, this required fixing how log.stream was aborted. There was a race condition or something like that. We need to set the reference to log.stream.abort before we start draining the log.stream.
1st :x: 2nd :heavy_check_mark: