ssbc / ssb-db2

A new database for secure-scuttlebutt
47 stars 8 forks source link

private index updates the status #400

Closed staltz closed 1 year ago

staltz commented 1 year ago

Context

https://github.com/ssbc/ssb-ebt/issues/77

When the private index alone was being rebuilt, no other index, then Manyverse would not show a progress bar.

Problem

Private index is actually not included in the status object that is used for calculating indexing progress.

Solution

Now it is.

1st :x: 2nd :heavy_check_mark:

github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 354.95ms
Validate 5000 messages 476.81ms
Native to db format 5000 messages 112.99ms
Db to native format 5000 messages 81.30ms
Add 1000 elements 371.46ms
Add 1000 box1 msgs 1001.18ms
Unbox 1000 box1 msgs first run 218.61ms
Unbox 1000 box1 msgs second run 102.12ms
Add 1000 box1 msgs 964.51ms
Query 1000 msgs first run 43.77ms
Query 1000 msgs second run 19.93ms
Add 1000 box2 msgs 1277.32ms
Unbox 1000 box2 msgs first run 338.83ms
Unbox 1000 box2 msgs second run 172.41ms
Migrate (+db1) 11053.37ms
Migrate (alone) 3540.17ms
Migrate (+db1 +db2) 7947.24ms
Migrate (+db2) 5538.66ms
Migrate continuation (+db2) 912.42ms
Memory usage without indexes 757.74 MB = 40.07 MB + etc
Initial indexing 1527.05ms
Initial indexing maxcpu=86 4892.95ms
Initial indexing compat 585.00ms
Two indexes updating concurrently 2770.13ms
Key one initial 47.03ms
Key two 1.27ms
Key one again 1.14ms
Reboot and key one again 54.36ms
Latest root posts 1668.37ms
Latest posts 11.92ms
Votes one initial 1448.10ms
Votes again 0.50ms
HasRoot 1290.78ms
HasRoot again 0.29ms
Author one posts 1477.58ms
Author two posts 18.40ms
Dedicated author one posts 1427.51ms
Dedicated author one posts again 0.47ms
DeleteFeed 2527.49ms
Maximum memory usage 983.40 MB = 63.15 MB + etc
Indexes folder size 10.01mb
github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 462.25ms
Validate 5000 messages 531.98ms
Native to db format 5000 messages 123.82ms
Db to native format 5000 messages 116.48ms
Add 1000 elements 383.80ms
Add 1000 box1 msgs 1144.19ms
Unbox 1000 box1 msgs first run 248.07ms
Unbox 1000 box1 msgs second run 124.65ms
Add 1000 box1 msgs 1103.57ms
Query 1000 msgs first run 57.20ms
Query 1000 msgs second run 13.67ms
Add 1000 box2 msgs 1637.02ms
Unbox 1000 box2 msgs first run 461.34ms
Unbox 1000 box2 msgs second run 213.33ms
Migrate (+db1) 15110.59ms
Migrate (alone) 4904.95ms
Migrate (+db1 +db2) 10328.86ms
Migrate (+db2) 7514.32ms
Migrate continuation (+db2) 1138.31ms
Memory usage without indexes 745.05 MB = 40.64 MB + etc
Initial indexing 1890.67ms
Initial indexing maxcpu=86 6330.70ms
Initial indexing compat 947.89ms
Two indexes updating concurrently 3549.73ms
Key one initial 56.87ms
Key two 3.15ms
Key one again 1.50ms
Reboot and key one again 53.75ms
Latest root posts 2218.09ms
Latest posts 11.32ms
Votes one initial 1782.18ms
Votes again 0.68ms
HasRoot 1638.89ms
HasRoot again 0.42ms
Author one posts 1766.02ms
Author two posts 21.56ms
Dedicated author one posts 1741.63ms
Dedicated author one posts again 0.97ms
DeleteFeed 3348.74ms
Maximum memory usage 1001.62 MB = 58.80 MB + etc
Indexes folder size 10.01mb