github-actions[bot]
commented
1 year ago Benchmark results
| Part | Duration |
|---|---|
| Create 5000 new messages | 369.37ms |
| Validate 5000 messages | 484.17ms |
| Native to db format 5000 messages | 108.54ms |
| Db to native format 5000 messages | 82.56ms |
| Add 1000 elements | 495.09ms |
| Add 1000 box1 msgs | 1005.79ms |
| Unbox 1000 box1 msgs first run | 222.32ms |
| Unbox 1000 box1 msgs second run | 105.55ms |
| Add 1000 box1 msgs | 973.86ms |
| Query 1000 msgs first run | 38.73ms |
| Query 1000 msgs second run | 13.19ms |
| Add 1000 box2 msgs | 1296.46ms |
| Unbox 1000 box2 msgs first run | 341.27ms |
| Unbox 1000 box2 msgs second run | 169.88ms |
| Migrate (+db1) | 11206.96ms |
| Migrate (alone) | 3759.22ms |
| Migrate (+db1 +db2) | 8326.04ms |
| Migrate (+db2) | 5807.76ms |
| Migrate continuation (+db2) | 884.10ms |
| Memory usage without indexes | 742.67 MB = 39.72 MB + etc |
| Initial indexing | 1602.47ms |
| Initial indexing maxcpu=86 | 5109.61ms |
| Initial indexing compat | 690.63ms |
| Two indexes updating concurrently | 2864.10ms |
| Key one initial | 44.60ms |
| Key two | 0.51ms |
| Key one again | 1.48ms |
| Reboot and key one again | 55.70ms |
| Latest root posts | 1706.27ms |
| Latest posts | 12.23ms |
| Votes one initial | 1496.52ms |
| Votes again | 0.48ms |
| HasRoot | 1304.02ms |
| HasRoot again | 0.32ms |
| Author one posts | 1457.27ms |
| Author two posts | 35.46ms |
| Dedicated author one posts | 1464.60ms |
| Dedicated author one posts again | 0.53ms |
| DeleteFeed | 2757.82ms |
| Maximum memory usage | 969.16 MB = 58.58 MB + etc |
| Indexes folder size | 10.01mb |
Context
https://github.com/ssbc/ssb-ebt/issues/77
Problem
multicb is a bit tricky to use. It can generate CBs with
done(), but if you calldone(() => {...then it will wait for all the generated CBs to get called.Problem here is that we were generating the CBs too late, we were generating them after the file had loaded, which meant that when it proceeded to
done((criticalError, results) => {it was NOT waiting for files to be loaded, and thus we would always get offset-1for these indexes, which would cause a full rebuild of them.Solution
Generate the multicb callbacks synchronously, before waiting for them.
Unfortunately no tests added because I'm confused as to how to test this. We don't have a proper hook to catch the moment when the private index is loaded.
onDrainisn't that because it's going to wait for the private index to be built. And the base index and private indexes are built eagerly ASAP, so :shrug: