ssbc / ssb-db2

A new database for secure-scuttlebutt
47 stars 8 forks source link

add tests for over-sized classic msgs, patch #394

Closed mixmix closed 1 year ago

mixmix commented 1 year ago

Leans on https://github.com/ssbc/ssb-classic/pull/2

This adds tests for over-sized classic messages (plaintext + encrypted).

TODO:

github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 514.38ms
Validate 5000 messages 587.94ms
Native to db format 5000 messages 137.92ms
Db to native format 5000 messages 104.20ms
Add 1000 elements 519.55ms
Add 1000 box1 msgs 1227.01ms
Unbox 1000 box1 msgs first run 303.70ms
Unbox 1000 box1 msgs second run 152.23ms
Add 1000 box1 msgs 1203.91ms
Query 1000 msgs first run 58.07ms
Query 1000 msgs second run 15.16ms
Add 1000 box2 msgs 1806.26ms
Unbox 1000 box2 msgs first run 451.45ms
Unbox 1000 box2 msgs second run 204.96ms
Migrate (+db1) 15585.95ms
Migrate (alone) 5413.05ms
Migrate (+db1 +db2) 10924.43ms
Migrate (+db2) 7749.73ms
Migrate continuation (+db2) 1350.59ms
Memory usage without indexes 756.56 MB = 38.89 MB + etc
Initial indexing 1989.17ms
Initial indexing maxcpu=86 6511.50ms
Initial indexing compat 1106.65ms
Two indexes updating concurrently 3691.12ms
Key one initial 51.64ms
Key two 1.53ms
Key one again 1.70ms
Reboot and key one again 52.87ms
Latest root posts 2151.07ms
Latest posts 10.85ms
Votes one initial 1887.66ms
Votes again 0.67ms
HasRoot 1684.29ms
HasRoot again 0.36ms
Author one posts 1784.11ms
Author two posts 28.75ms
Dedicated author one posts 1780.98ms
Dedicated author one posts again 0.92ms
DeleteFeed 3824.22ms
Maximum memory usage 994.30 MB = 58.87 MB + etc
Indexes folder size 10.01mb
staltz commented 1 year ago

I converted this PR to draft just to signal that it can't be merged before

install ssb-classic correctly once that's released

github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 359.43ms
Validate 5000 messages 450.88ms
Native to db format 5000 messages 114.77ms
Db to native format 5000 messages 83.23ms
Add 1000 elements 492.73ms
Add 1000 box1 msgs 941.40ms
Unbox 1000 box1 msgs first run 212.86ms
Unbox 1000 box1 msgs second run 104.74ms
Add 1000 box1 msgs 899.98ms
Query 1000 msgs first run 45.67ms
Query 1000 msgs second run 22.18ms
Add 1000 box2 msgs 1286.77ms
Unbox 1000 box2 msgs first run 312.48ms
Unbox 1000 box2 msgs second run 170.05ms
Migrate (+db1) 11487.80ms
Migrate (alone) 4014.71ms
Migrate (+db1 +db2) 8380.23ms
Migrate (+db2) 6102.18ms
Migrate continuation (+db2) 861.66ms
Memory usage without indexes 743.95 MB = 43.55 MB + etc
Initial indexing 1689.56ms
Initial indexing maxcpu=86 4991.78ms
Initial indexing compat 916.29ms
Two indexes updating concurrently 2947.52ms
Key one initial 46.79ms
Key two 0.56ms
Key one again 1.87ms
Reboot and key one again 45.12ms
Latest root posts 1753.58ms
Latest posts 11.68ms
Votes one initial 1478.95ms
Votes again 0.57ms
HasRoot 1293.78ms
HasRoot again 0.36ms
Author one posts 1423.00ms
Author two posts 19.09ms
Dedicated author one posts 1410.23ms
Dedicated author one posts again 0.72ms
DeleteFeed 2955.47ms
Maximum memory usage 990.76 MB = 60.83 MB + etc
Indexes folder size 10.01mb
Powersource commented 1 year ago

@staltz can you publish a new version?

github-actions[bot] commented 1 year ago

Benchmark results

Part Duration
Create 5000 new messages 458.28ms
Validate 5000 messages 528.23ms
Native to db format 5000 messages 141.98ms
Db to native format 5000 messages 102.43ms
Add 1000 elements 512.22ms
Add 1000 box1 msgs 1105.11ms
Unbox 1000 box1 msgs first run 248.52ms
Unbox 1000 box1 msgs second run 124.92ms
Add 1000 box1 msgs 1062.72ms
Query 1000 msgs first run 59.37ms
Query 1000 msgs second run 16.24ms
Add 1000 box2 msgs 1503.57ms
Unbox 1000 box2 msgs first run 402.24ms
Unbox 1000 box2 msgs second run 209.80ms
Migrate (+db1) 13216.82ms
Migrate (alone) 4974.24ms
Migrate (+db1 +db2) 9961.04ms
Migrate (+db2) 7211.87ms
Migrate continuation (+db2) 1117.86ms
Memory usage without indexes 749.77 MB = 40.02 MB + etc
Initial indexing 1984.86ms
Initial indexing maxcpu=86 5878.22ms
Initial indexing compat 953.44ms
Two indexes updating concurrently 3333.58ms
Key one initial 53.20ms
Key two 1.88ms
Key one again 2.53ms
Reboot and key one again 49.15ms
Latest root posts 2079.95ms
Latest posts 13.39ms
Votes one initial 1727.80ms
Votes again 0.60ms
HasRoot 1503.63ms
HasRoot again 0.39ms
Author one posts 1670.86ms
Author two posts 20.49ms
Dedicated author one posts 1695.96ms
Dedicated author one posts again 0.56ms
DeleteFeed 3662.29ms
Maximum memory usage 1009.23 MB = 64.26 MB + etc
Indexes folder size 10.01mb
staltz commented 1 year ago

Published ssb-db@6.2.3