ssbc / ssb-db2

A new database for secure-scuttlebutt
47 stars 8 forks source link

Simplify deleteFeed #403

Closed arj03 closed 1 year ago

arj03 commented 1 year ago

This fixes #402. No need to load up all the offsets into memory if we just delete all of them anyway.

github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 368.50ms
Validate 5000 messages 448.87ms
Native to db format 5000 messages 133.31ms
Db to native format 5000 messages 87.13ms
Add 1000 elements 522.64ms
Add 1000 box1 msgs 937.23ms
Unbox 1000 box1 msgs first run 220.98ms
Unbox 1000 box1 msgs second run 103.01ms
Add 1000 box1 msgs 904.02ms
Query 1000 msgs first run 39.47ms
Query 1000 msgs second run 22.98ms
Add 1000 box2 msgs 1297.41ms
Unbox 1000 box2 msgs first run 339.62ms
Unbox 1000 box2 msgs second run 151.35ms
Migrate (+db1) 11840.34ms
Migrate (alone) 3979.39ms
Migrate (+db1 +db2) 8259.00ms
Migrate (+db2) 6196.09ms
Migrate continuation (+db2) 1204.59ms
Memory usage without indexes 757.43 MB = 37.74 MB + etc
Initial indexing 1735.55ms
Initial indexing maxcpu=86 5028.50ms
Initial indexing compat 677.80ms
Two indexes updating concurrently 2947.23ms
Key one initial 53.16ms
Key two 0.53ms
Key one again 1.96ms
Reboot and key one again 43.37ms
Latest root posts 1741.00ms
Latest posts 10.23ms
Votes one initial 1516.96ms
Votes again 0.59ms
HasRoot 1305.77ms
HasRoot again 0.31ms
Author one posts 1427.55ms
Author two posts 15.99ms
Dedicated author one posts 1449.55ms
Dedicated author one posts again 0.51ms
DeleteFeed 141854.89ms
Maximum memory usage 975.20 MB = 55.57 MB + etc
Indexes folder size 10.01mb
github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 365.83ms
Validate 5000 messages 447.78ms
Native to db format 5000 messages 114.84ms
Db to native format 5000 messages 82.26ms
Add 1000 elements 365.38ms
Add 1000 box1 msgs 930.05ms
Unbox 1000 box1 msgs first run 209.70ms
Unbox 1000 box1 msgs second run 106.42ms
Add 1000 box1 msgs 894.70ms
Query 1000 msgs first run 47.69ms
Query 1000 msgs second run 12.65ms
Add 1000 box2 msgs 1284.79ms
Unbox 1000 box2 msgs first run 316.14ms
Unbox 1000 box2 msgs second run 143.91ms
Migrate (+db1) 11830.30ms
Migrate (alone) 4061.75ms
Migrate (+db1 +db2) 7995.15ms
Migrate (+db2) 6448.79ms
Migrate continuation (+db2) 977.58ms
Memory usage without indexes 742.50 MB = 40.44 MB + etc
Initial indexing 1619.88ms
Initial indexing maxcpu=86 5357.09ms
Initial indexing compat 747.66ms
Two indexes updating concurrently 2735.11ms
Key one initial 54.74ms
Key two 0.55ms
Key one again 1.48ms
Reboot and key one again 45.59ms
Latest root posts 1845.36ms
Latest posts 8.52ms
Votes one initial 1501.42ms
Votes again 0.62ms
HasRoot 1315.32ms
HasRoot again 0.33ms
Author one posts 1446.41ms
Author two posts 19.34ms
Dedicated author one posts 1456.70ms
Dedicated author one posts again 0.51ms
DeleteFeed 139037.52ms
Maximum memory usage 928.03 MB = 63.38 MB + etc
Indexes folder size 10.01mb
arj03 commented 1 year ago

I can see that DeleteFeed seems to be much slower than before. I'll try giving it a look, but it is a bit strange.

github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 563.80ms
Validate 5000 messages 590.06ms
Native to db format 5000 messages 162.91ms
Db to native format 5000 messages 116.84ms
Add 1000 elements 521.55ms
Add 1000 box1 msgs 1372.09ms
Unbox 1000 box1 msgs first run 306.52ms
Unbox 1000 box1 msgs second run 136.78ms
Add 1000 box1 msgs 1355.47ms
Query 1000 msgs first run 53.03ms
Query 1000 msgs second run 26.39ms
Add 1000 box2 msgs 2029.51ms
Unbox 1000 box2 msgs first run 500.66ms
Unbox 1000 box2 msgs second run 243.05ms
Migrate (+db1) 16675.95ms
Migrate (alone) 5707.53ms
Migrate (+db1 +db2) 12727.93ms
Migrate (+db2) 8444.87ms
Migrate continuation (+db2) 1402.06ms
Memory usage without indexes 759.13 MB = 38.49 MB + etc
Initial indexing 2145.05ms
Initial indexing maxcpu=86 6365.95ms
Initial indexing compat 1176.11ms
Two indexes updating concurrently 3876.36ms
Key one initial 61.33ms
Key two 1.37ms
Key one again 1.44ms
Reboot and key one again 64.79ms
Latest root posts 2419.49ms
Latest posts 16.97ms
Votes one initial 2014.44ms
Votes again 0.75ms
HasRoot 1735.90ms
HasRoot again 1.11ms
Author one posts 1863.42ms
Author two posts 28.08ms
Dedicated author one posts 1901.61ms
Dedicated author one posts again 0.67ms
DeleteFeed 147365.05ms
Maximum memory usage 930.62 MB = 61.07 MB + etc
Indexes folder size 10.01mb
staltz commented 1 year ago

(Approved just in case it wasn't clear from discussion, but I know you might want to still work on this)

arj03 commented 1 year ago

I pushed up a batch(1000), then the performance is the same as before. We need the same here. I'll do that as seperate PR.

github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 372.92ms
Validate 5000 messages 483.66ms
Native to db format 5000 messages 108.28ms
Db to native format 5000 messages 83.85ms
Add 1000 elements 372.08ms
Add 1000 box1 msgs 1004.54ms
Unbox 1000 box1 msgs first run 217.13ms
Unbox 1000 box1 msgs second run 104.22ms
Add 1000 box1 msgs 975.55ms
Query 1000 msgs first run 40.71ms
Query 1000 msgs second run 13.26ms
Add 1000 box2 msgs 1309.02ms
Unbox 1000 box2 msgs first run 305.79ms
Unbox 1000 box2 msgs second run 139.82ms
Migrate (+db1) 11382.09ms
Migrate (alone) 4054.34ms
Migrate (+db1 +db2) 8455.81ms
Migrate (+db2) 5889.11ms
Migrate continuation (+db2) 885.89ms
Memory usage without indexes 753.36 MB = 40.53 MB + etc
Initial indexing 1591.65ms
Initial indexing maxcpu=86 4839.90ms
Initial indexing compat 705.78ms
Two indexes updating concurrently 2839.31ms
Key one initial 52.21ms
Key two 0.50ms
Key one again 1.12ms
Reboot and key one again 42.67ms
Latest root posts 1719.27ms
Latest posts 9.31ms
Votes one initial 1459.52ms
Votes again 0.59ms
HasRoot 1316.85ms
HasRoot again 0.56ms
Author one posts 1407.09ms
Author two posts 21.13ms
Dedicated author one posts 1429.79ms
Dedicated author one posts again 0.58ms
DeleteFeed 5184.24ms
Maximum memory usage 996.18 MB = 59.98 MB + etc
Indexes folder size 10.01mb
github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 361.74ms
Validate 5000 messages 481.48ms
Native to db format 5000 messages 108.95ms
Db to native format 5000 messages 81.77ms
Add 1000 elements 520.85ms
Add 1000 box1 msgs 997.72ms
Unbox 1000 box1 msgs first run 224.21ms
Unbox 1000 box1 msgs second run 107.33ms
Add 1000 box1 msgs 970.64ms
Query 1000 msgs first run 35.68ms
Query 1000 msgs second run 19.94ms
Add 1000 box2 msgs 1290.31ms
Unbox 1000 box2 msgs first run 314.83ms
Unbox 1000 box2 msgs second run 145.18ms
Migrate (+db1) 10889.34ms
Migrate (alone) 3636.71ms
Migrate (+db1 +db2) 8113.63ms
Migrate (+db2) 5693.87ms
Migrate continuation (+db2) 841.02ms
Memory usage without indexes 745.05 MB = 41.42 MB + etc
Initial indexing 1559.67ms
Initial indexing maxcpu=86 4950.33ms
Initial indexing compat 715.86ms
Two indexes updating concurrently 2770.41ms
Key one initial 45.96ms
Key two 1.36ms
Key one again 0.78ms
Reboot and key one again 49.91ms
Latest root posts 1673.38ms
Latest posts 9.06ms
Votes one initial 1459.72ms
Votes again 0.49ms
HasRoot 1276.52ms
HasRoot again 0.28ms
Author one posts 1389.99ms
Author two posts 14.63ms
Dedicated author one posts 1442.78ms
Dedicated author one posts again 0.51ms
DeleteFeed 4954.22ms
Maximum memory usage 1004.93 MB = 61.59 MB + etc
Indexes folder size 10.01mb