add publicWebHosting to aboutSelf

mixmix commented 2 years ago

Problem:

Planetary viewer system uses ssb-db2 to serve threads, profiles over the web
The current indexes/about-self don't track publicWebHosting

This PR:

adds support for this to the aboutSelf index
adds isString checks to name and description fields
- @staltz in Mix's head "make it a different PR"
- @mixmix "no, this has to be here, it's not safe without this check"
- @arj03 "actually it's faster if you just chain heaps of if &&, what you've written is not easy for v8 to optimise"
- @mixmix :sob:

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	365.90ms
Validate 5000 messages	450.98ms
Native to db format 5000 messages	116.90ms
Db to native format 5000 messages	87.91ms
Add 1000 elements	492.76ms
Add 1000 box1 msgs	923.68ms
Unbox 1000 box1 msgs first run	211.44ms
Unbox 1000 box1 msgs second run	105.49ms
Add 1000 box1 msgs	891.25ms
Query 1000 msgs first run	44.21ms
Query 1000 msgs second run	20.63ms
Add 1000 box2 msgs	1267.70ms
Unbox 1000 box2 msgs first run	317.11ms
Unbox 1000 box2 msgs second run	145.60ms
Migrate (+db1)	11800.40ms
Migrate (alone)	4341.83ms
Migrate (+db1 +db2)	8071.70ms
Migrate (+db2)	6553.32ms
Migrate continuation (+db2)	959.74ms
Memory usage without indexes	746.37 MB = 39.76 MB + etc
Initial indexing	1622.36ms
Initial indexing maxcpu=86	5007.01ms
Initial indexing compat	637.70ms
Two indexes updating concurrently	2877.53ms
Key one initial	44.86ms
Key two	1.20ms
Key one again	0.86ms
Reboot and key one again	45.97ms
Latest root posts	1719.30ms
Latest posts	8.92ms
Votes one initial	1450.97ms
Votes again	0.53ms
HasRoot	1329.18ms
HasRoot again	0.31ms
Author one posts	1387.03ms
Author two posts	23.89ms
Dedicated author one posts	1414.71ms
Dedicated author one posts again	0.58ms
DeleteFeed	3241.98ms
Maximum memory usage	966.89 MB = 58.47 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	370.65ms
Validate 5000 messages	447.78ms
Native to db format 5000 messages	114.41ms
Db to native format 5000 messages	86.26ms
Add 1000 elements	365.70ms
Add 1000 box1 msgs	925.62ms
Unbox 1000 box1 msgs first run	207.38ms
Unbox 1000 box1 msgs second run	104.80ms
Add 1000 box1 msgs	894.45ms
Query 1000 msgs first run	49.75ms
Query 1000 msgs second run	12.69ms
Add 1000 box2 msgs	1263.77ms
Unbox 1000 box2 msgs first run	320.55ms
Unbox 1000 box2 msgs second run	146.34ms
Migrate (+db1)	11084.68ms
Migrate (alone)	3754.78ms
Migrate (+db1 +db2)	7829.66ms
Migrate (+db2)	5899.25ms
Migrate continuation (+db2)	916.70ms
Memory usage without indexes	744.04 MB = 39.89 MB + etc
Initial indexing	1586.59ms
Initial indexing maxcpu=86	4917.83ms
Initial indexing compat	646.04ms
Two indexes updating concurrently	2862.34ms
Key one initial	41.08ms
Key two	0.92ms
Key one again	0.53ms
Reboot and key one again	47.15ms
Latest root posts	1705.18ms
Latest posts	9.30ms
Votes one initial	1426.45ms
Votes again	0.49ms
HasRoot	1253.64ms
HasRoot again	0.30ms
Author one posts	1340.21ms
Author two posts	15.02ms
Dedicated author one posts	1399.88ms
Dedicated author one posts again	0.59ms
DeleteFeed	2776.34ms
Maximum memory usage	971.13 MB = 56.95 MB + etc
Indexes folder size	10.01mb

staltz commented 2 years ago

Hey @mixmix, some time ago I extracted this plugin out of ssb-db2 and put it at https://github.com/staltz/ssb-about-self. We're going to remove it from ssb-db2, so it's not worth updating it here. Can you make the PR over there?

mixmix commented 2 years ago

Yeah sure thing. I'll likely make a pr which changes this repo to import that module and export it so there's no divergence of logic

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	367.32ms
Validate 5000 messages	449.46ms
Native to db format 5000 messages	112.83ms
Db to native format 5000 messages	86.97ms
Add 1000 elements	487.63ms
Add 1000 box1 msgs	912.49ms
Unbox 1000 box1 msgs first run	209.61ms
Unbox 1000 box1 msgs second run	105.27ms
Add 1000 box1 msgs	886.93ms
Query 1000 msgs first run	53.79ms
Query 1000 msgs second run	12.30ms
Add 1000 box2 msgs	1264.26ms
Unbox 1000 box2 msgs first run	354.14ms
Unbox 1000 box2 msgs second run	169.18ms
Migrate (+db1)	11382.36ms
Migrate (alone)	3989.47ms
Migrate (+db1 +db2)	8047.29ms
Migrate (+db2)	6150.10ms
Migrate continuation (+db2)	874.32ms
Memory usage without indexes	751.90 MB = 42.71 MB + etc
Initial indexing	1651.95ms
Initial indexing maxcpu=86	4974.88ms
Initial indexing compat	760.43ms
Two indexes updating concurrently	2817.31ms
Key one initial	45.09ms
Key two	0.53ms
Key one again	1.32ms
Reboot and key one again	51.96ms
Latest root posts	1702.73ms
Latest posts	9.76ms
Votes one initial	1481.84ms
Votes again	0.51ms
HasRoot	1276.55ms
HasRoot again	0.36ms
Author one posts	1407.76ms
Author two posts	35.25ms
Dedicated author one posts	1402.15ms
Dedicated author one posts again	0.50ms
DeleteFeed	2803.60ms
Maximum memory usage	1001.89 MB = 61.13 MB + etc
Indexes folder size	10.01mb

mixmix commented 2 years ago

@staltz can you review this again please?

staltz commented 1 year ago

@mixmix I have, but you haven't apparently seen https://github.com/ssbc/ssb-db2/pull/388#discussion_r969434521

github-actions[bot] commented 1 year ago

Benchmark results

Part	Duration
Create 5000 new messages	359.84ms
Validate 5000 messages	484.04ms
Native to db format 5000 messages	108.60ms
Db to native format 5000 messages	81.99ms
Add 1000 elements	367.58ms
Add 1000 box1 msgs	1002.19ms
Unbox 1000 box1 msgs first run	216.31ms
Unbox 1000 box1 msgs second run	103.30ms
Add 1000 box1 msgs	963.64ms
Query 1000 msgs first run	39.44ms
Query 1000 msgs second run	20.16ms
Add 1000 box2 msgs	1287.69ms
Unbox 1000 box2 msgs first run	326.16ms
Unbox 1000 box2 msgs second run	156.47ms
Migrate (+db1)	11144.17ms
Migrate (alone)	3600.04ms
Migrate (+db1 +db2)	7973.75ms
Migrate (+db2)	5773.91ms
Migrate continuation (+db2)	844.15ms
Memory usage without indexes	737.13 MB = 37.55 MB + etc
Initial indexing	1567.06ms
Initial indexing maxcpu=86	4113.72ms
Initial indexing compat	677.97ms
Two indexes updating concurrently	2743.71ms
Key one initial	56.27ms
Key two	1.91ms
Key one again	1.87ms
Reboot and key one again	41.73ms
Latest root posts	1670.94ms
Latest posts	9.38ms
Votes one initial	1459.27ms
Votes again	0.54ms
HasRoot	1310.90ms
HasRoot again	0.29ms
Author one posts	1410.32ms
Author two posts	14.76ms
Dedicated author one posts	1421.99ms
Dedicated author one posts again	0.49ms
DeleteFeed	2850.50ms
Maximum memory usage	1008.64 MB = 61.93 MB + etc
Indexes folder size	10.01mb

staltz commented 1 year ago

Merged because you said in a private chat that

I would have preferred you just committed that, merged and published.

staltz commented 1 year ago

v6.2.1

github-actions[bot] commented 1 year ago

Benchmark results

Part	Duration
Create 5000 new messages	622.26ms
Validate 5000 messages	671.75ms
Native to db format 5000 messages	156.31ms
Db to native format 5000 messages	108.24ms
Add 1000 elements	539.28ms
Add 1000 box1 msgs	1332.67ms
Unbox 1000 box1 msgs first run	311.29ms
Unbox 1000 box1 msgs second run	150.59ms
Add 1000 box1 msgs	1276.49ms
Query 1000 msgs first run	52.47ms
Query 1000 msgs second run	26.04ms
Add 1000 box2 msgs	1853.87ms
Unbox 1000 box2 msgs first run	484.73ms
Unbox 1000 box2 msgs second run	258.23ms
Migrate (+db1)	15975.14ms
Migrate (alone)	5764.03ms
Migrate (+db1 +db2)	12259.67ms
Migrate (+db2)	8059.09ms
Migrate continuation (+db2)	1327.45ms
Memory usage without indexes	763.18 MB = 35.15 MB + etc
Initial indexing	2020.21ms
Initial indexing maxcpu=86	6322.19ms
Initial indexing compat	1179.48ms
Two indexes updating concurrently	3727.31ms
Key one initial	61.19ms
Key two	0.61ms
Key one again	1.46ms
Reboot and key one again	69.35ms
Latest root posts	2300.48ms
Latest posts	13.23ms
Votes one initial	1879.07ms
Votes again	0.69ms
HasRoot	1744.39ms
HasRoot again	0.41ms
Author one posts	1891.60ms
Author two posts	20.81ms
Dedicated author one posts	1835.13ms
Dedicated author one posts again	0.70ms
DeleteFeed	3674.21ms
Maximum memory usage	1007.64 MB = 61.76 MB + etc
Indexes folder size	10.01mb

ssbc / ssb-db2