github-actions[bot]
commented
2 years ago Benchmark results
| Part | Duration |
|---|---|
| add 1000 elements | 494ms |
| add 1000 private box1 elements | 1155ms |
| unbox 1000 private box1 elements first run | 125ms |
| unbox 1000 private box1 elements second run | 104ms |
| add 1000 private box1 elements | 1141ms |
| query 1000 elements first run | 28ms |
| query 1000 elements second run | 13ms |
| add 1000 private box2 elements | 634ms |
| unbox 1000 private box2 elements first run | 361ms |
| unbox 1000 private box2 elements second run | 355ms |
| Migrate (+db1) | 11704ms |
| Migrate (alone) | 4097ms |
| Migrate (+db1 +db2) | 8059ms |
| Migrate (+db2) | 6684ms |
| Migrate continuation (+db2) | 930ms |
| Memory usage without indexes | 739.18 MB = 32.26 MB + etc |
| Initial indexing | 693ms |
| Initial indexing maxCpu=86 | 4572ms |
| Initial indexing compat | 900ms |
| Two indexes updating concurrently | 891ms |
| key one initial | 44ms |
| key two | 0ms |
| key one again | 1ms |
| reboot and key one again | 45ms |
| latest root posts | 633ms |
| latest posts | 9ms |
| votes one initial | 552ms |
| votes again | 0ms |
| hasRoot | 395ms |
| hasRoot again | 0ms |
| author one posts | 296ms |
| author two posts | 18ms |
| dedicated author one posts | 330ms |
| dedicated author one posts again | 1ms |
| Maximum memory usage | 880.72 MB = 55.14 MB + etc |
| Indexes folder size | 9.97mb |
Context
When running deleteFeeds-then-compact-then-reindex experiments in Manyverse, I noticed that the Activity tab highlighted as "new", which didn't make sense because there was no new content added (replication was turned off).
This happened consistently, regardless of what dataset I was using (production or ssb-fixtures).
See also https://github.com/ssbc/ssb-db2/issues/306#issuecomment-1161950044
Problem
It turned out the problem was with leveldb live. The way that
pull-levelandlevel-postwork is by adding listeners to the leveldb EventEmitter'put','del','batch': https://github.com/dominictarr/level-post/blob/9021f153f159aa1d13233908ebe616f650e78f22/index.js#L44-L46Obviously, during reindexing, we use
batchto recreate the index, and this triggers the "live" side of pull-level, which then triggers fullMentions live stream. The Activity tab in Manyverse uses fullMentions. But I also think that any other leveldb index which uses live, such as aboutSelf, would be affected in a similar way.Solution
Not the cleanest solution, but I wanted to find a solution that is general for all our leveldb indexes AND contained to ssb-db2 repo to not require each index to be updated (which is error prone).
I figured that we can just rip out all the EventEmitter listeners when the leveldb index is "reset" (which so far ONLY happens after compaction), and then wait until the index catches up with the full log, and then put those listeners back in.
1st :x: 2nd :heavy_check_mark: