ssbc / ssb-db2

A new database for secure-scuttlebutt
47 stars 8 forks source link

Batch 1000 to speed up reindexEncrypted #405

Closed arj03 closed 1 year ago

arj03 commented 1 year ago

Similar to #403

github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 539.09ms
Validate 5000 messages 593.00ms
Native to db format 5000 messages 159.76ms
Db to native format 5000 messages 109.00ms
Add 1000 elements 376.49ms
Add 1000 box1 msgs 1310.56ms
Unbox 1000 box1 msgs first run 271.44ms
Unbox 1000 box1 msgs second run 112.40ms
Add 1000 box1 msgs 1234.11ms
Query 1000 msgs first run 32.12ms
Query 1000 msgs second run 21.35ms
Add 1000 box2 msgs 1920.49ms
Unbox 1000 box2 msgs first run 402.95ms
Unbox 1000 box2 msgs second run 185.45ms
Migrate (+db1) 16124.40ms
Migrate (alone) 5624.28ms
Migrate (+db1 +db2) 11026.28ms
Migrate (+db2) 8668.08ms
Migrate continuation (+db2) 1265.31ms
Memory usage without indexes 738.33 MB = 40.36 MB + etc
Initial indexing 1869.75ms
Initial indexing maxcpu=86 6346.90ms
Initial indexing compat 921.78ms
Two indexes updating concurrently 3368.50ms
Key one initial 53.24ms
Key two 0.86ms
Key one again 1.81ms
Reboot and key one again 66.10ms
Latest root posts 2088.40ms
Latest posts 14.75ms
Votes one initial 1683.28ms
Votes again 1.00ms
HasRoot 1470.30ms
HasRoot again 0.36ms
Author one posts 1601.61ms
Author two posts 16.82ms
Dedicated author one posts 1593.38ms
Dedicated author one posts again 0.45ms
DeleteFeed 6833.31ms
Maximum memory usage 1007.53 MB = 55.70 MB + etc
Indexes folder size 10.01mb
arj03 commented 1 year ago

This should help a lot in tribes2 :)

github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 352.81ms
Validate 5000 messages 451.69ms
Native to db format 5000 messages 116.31ms
Db to native format 5000 messages 85.59ms
Add 1000 elements 491.11ms
Add 1000 box1 msgs 931.85ms
Unbox 1000 box1 msgs first run 212.15ms
Unbox 1000 box1 msgs second run 101.28ms
Add 1000 box1 msgs 893.92ms
Query 1000 msgs first run 37.42ms
Query 1000 msgs second run 22.77ms
Add 1000 box2 msgs 1280.72ms
Unbox 1000 box2 msgs first run 335.74ms
Unbox 1000 box2 msgs second run 154.45ms
Migrate (+db1) 11656.89ms
Migrate (alone) 4178.51ms
Migrate (+db1 +db2) 8165.17ms
Migrate (+db2) 6352.80ms
Migrate continuation (+db2) 1078.37ms
Memory usage without indexes 744.74 MB = 39.48 MB + etc
Initial indexing 1666.80ms
Initial indexing maxcpu=86 5387.37ms
Initial indexing compat 736.42ms
Two indexes updating concurrently 2739.29ms
Key one initial 49.82ms
Key two 2.42ms
Key one again 1.13ms
Reboot and key one again 41.47ms
Latest root posts 1777.68ms
Latest posts 9.34ms
Votes one initial 1476.57ms
Votes again 0.60ms
HasRoot 1310.60ms
HasRoot again 0.31ms
Author one posts 1393.18ms
Author two posts 18.88ms
Dedicated author one posts 1421.20ms
Dedicated author one posts again 0.61ms
DeleteFeed 5492.74ms
Maximum memory usage 932.73 MB = 59.50 MB + etc
Indexes folder size 10.01mb
github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 375.71ms
Validate 5000 messages 456.20ms
Native to db format 5000 messages 117.70ms
Db to native format 5000 messages 86.67ms
Add 1000 elements 491.12ms
Add 1000 box1 msgs 939.49ms
Unbox 1000 box1 msgs first run 211.35ms
Unbox 1000 box1 msgs second run 99.81ms
Add 1000 box1 msgs 893.76ms
Query 1000 msgs first run 40.31ms
Query 1000 msgs second run 21.99ms
Add 1000 box2 msgs 1288.83ms
Unbox 1000 box2 msgs first run 293.62ms
Unbox 1000 box2 msgs second run 161.01ms
Migrate (+db1) 12750.68ms
Migrate (alone) 4452.45ms
Migrate (+db1 +db2) 7931.69ms
Migrate (+db2) 7027.94ms
Migrate continuation (+db2) 1008.24ms
Memory usage without indexes 748.23 MB = 33.20 MB + etc
Initial indexing 1664.87ms
Initial indexing maxcpu=86 5180.87ms
Initial indexing compat 754.95ms
Two indexes updating concurrently 2822.62ms
Key one initial 52.56ms
Key two 0.64ms
Key one again 0.33ms
Reboot and key one again 51.99ms
Latest root posts 1803.34ms
Latest posts 9.43ms
Votes one initial 1481.66ms
Votes again 0.60ms
HasRoot 1290.94ms
HasRoot again 0.33ms
Author one posts 1404.04ms
Author two posts 20.15ms
Dedicated author one posts 1425.36ms
Dedicated author one posts again 0.89ms
DeleteFeed 6588.09ms
Maximum memory usage 998.06 MB = 59.72 MB + etc
Indexes folder size 10.01mb