staltz commented 2 years ago

Context

We are adding buttwoo as a feed format, and it works differently than other formats, so we would introduce a lot of complexity. At the same time, we need to improve the situation of support for box2 before we move on with "ssb-tribes2" on ssb-db2.

Solution

sbot.db.installFeedFormat() and sbot.db.installEncryptionFormat() and encodings.

These are now orthogonal concerns, so any encryption format can work with any feed format, which can support different encodings.

:large_blue_circle: Encoding

An "encoding" is a way of representing a feed message. This PR supports two encodings: 'js' (JavaScript objects) and 'bipf' (BIPF buffers).

:yellow_circle: Feed Format

A feed format defines how to create messages that follow a schema and a spec. Every feed format is a plugin-like object with:

name string
encodings array of supported encoding names
functions to create and convert "native messages"
- newNativeMsg(opts)
- toNativeMsg(msgVal, encoding)
- fromNativeMsg(nativeMsg, encoding)
- fromDecryptedNativeMsg(plaintextBuf, nativeMsg, encoding)
encryption-related function
- toPlaintextBuffer(opts)
helper functions getX() and isX()
- getFeedId(nativeMsg)
- getMsgId(nativeMsg)
- isNativeMsg(x)
- isAuthor(author)
validation functions
- validateSingle()
- validateBatch()
- validateOOOBatch()

:star: "Native Msg" this concept is new and important. Every feed format has a "native message format", which is a shape for the message that is specific to the feed format. The feed format fully controls its own "nativeMsg" shape, and it can be anything.

For "classic", a nativeMsg is the traditional JavaScript object msgVal.
For "bendybutt-v1", a nativeMsg is a BFE-and-bencoded buffer with a shape determined by bendy-butt-spec.
For "buttwoo-v1", a nativeMsg is a BFE-and-bipf buffer with a shape determined by buttwoo-spec.

:black_circle: Encryption Format

An encryption format specifies how to encrypt and decrypt JavaScript buffers intended for some specific recipients. Every encryption format is a plugin-like object with:

name string, also used for the suffix on the ciphertext
setup(config, cb) OPTIONAL
encrypt(plaintextBuf, opts) => ciphertextBuf
decrypt(ciphertextBuf, opts) => plaintextBuf

TODO

[x] private.js special treatments of box1 and box2
[x] Implement publish in terms of create()
[x] isPrivate(encryptionFormat) for querying canDecrypt
[x] Move post Obv to compat
[x] :wrench: feedFormat.getSequence() (needed for ebt)
[x] :detective: Is buttwoo performant with ssb-ebt?
[x] :detective: WeakMap optimizations in feed formats, from/to native msg
[x] :detective: Figure out box2 key management/addition
[x] :detective: jitdb operator for querying encrypted-box1 and encrypted-box2
[x] :wrench: Restore reindex-encrypted.js tests
[x] :wrench: ssb-db2 should wait for all encryption formats to load
[x] :wrench: Fix benchmarks
[x] :abc: Better name for encryptionFormat.getRecipients() => getEncryptionKeys()
[x] :abc: canDecrypt.index naming?
[x] :balance_scale: ssb-keyring LGPL-3.0 release
[x] :rocket: Publish ssb-feed-format
[x] :rocket: Publish ssb-encryption-format
[x] :rocket: Publish ssb-classic
[x] :rocket: Breaking change ssb-bendy-butt
[x] :rocket: Publish ssb-box1
[x] :wrench: ssb-uri-spec buttwoo URI needs parent
[x] :wrench: ssb-uri2 buttwoo URI needs parent
[x] :rocket: Breaking change ssb-buttwoo
[x] :book: Document new APIs such as:
- create()
- onMsgAdded()
- installFeedFormat()
- installEncryptionFormat()
[x] :book: Document breaking changes such as:
- encrypted.index/canDecrypt.index
- setPost hack for partial replication in the browser
- isPrivate() => isDecrypted() / isEncrypted()
[x] :wrench: ssb-keyring changes to support derived-keys-only workflow
[x] :rocket: Publish ssb-box2
[x] MERGE AND PUBLISH VERSION 5.0.0
[x] ssb-ebt Replace post with onMsgAdded
[x] ssb-conn Replace post with onMsgAdded
[x] ssb-meta-feeds Update ssb-bendy-butt to new feed format API
[x] ssb-subset-ql Update with new db2 5.0.0

staltz commented 2 years ago

@arj03 All tests pass except the ones related to box2 simply because of key management that I haven't worked on yet. But box2 is supported and you can see that from test/create.js.

To use this in ssb-ebt for buttwoo, you'll have to do something like:

db => network

sbot.getAtSequenceNativeMsg(arr, (err, nativeMsg) => (no need to convert, just directly pass the nativeMsg to the network peer)

network => db

sbot.db.add(nativeMsg, {encoding: 'bipf', feedFormat: 'buttwoo'}, cb)
OR
sbot.db.add(nativeMsg, {encoding: 'bipf'}, cb) (might be just slightly slower than the above)

If you need some other data, let me know.

staltz commented 2 years ago

@arj03 Check out an internal refactor I figured out: https://github.com/ssbc/ssb-db2/pull/347/commits/4a8eb83a56e12aae75ef3d2a69db7e076000e2c8

staltz commented 2 years ago

Also, https://github.com/ssbc/ssb-db2/pull/347/commits/547df2537feea39c56b5a6e03617f7898dbebfee may look interesting, it creates the files

decrypted.index (former canDecrypt.index)
encrypted-box1.index (new)
encrypted-box2.index (former encrypted.index)

If you add a new encryption format named foo, it will create the file encrypted-foo.index.

mixmix commented 2 years ago

thoughts

what is getRecipients(opts) ?
what are recipients ? e.g. do you mean [%groupId.cloaked, @mixFeedId,]
are we using msgId or msgKey it low key annoys me that ssb flip-flops between these. Lets fix it!

I think this looks good but I find it quite hard to get a feel for it without visualising these in use? Maybe a conversation with that new pipeline diagram and pointing at the different parts and then naming the functions used would help contextualise.

Also I assume some of the function signatures are not really pinned, because I assume things like isAuthor(author) might be like isAuthor(nativeMsg, author), otherwise I've misunderstood

mixmix commented 2 years ago

Nice work btw

:horse_racing: :lizard: :melon:

staltz commented 2 years ago

Hey @mixmix thanks for taking a look at this:

are we using msgId or msgKey it low key annoys me that ssb flip-flops between these. Lets fix it!

In this PR I'm using msgId, and I can try to be consistent with that elsewhere as well. msgKey is ambiguous because in so many places we use the word "keys" for "cryptographic key pair". In fact I think I bumped into one box2 related module that had msgKey as a variable name meaning "read key of a box2 boxed msg". So I'm trying to avoid that ambiguity.

The exception, which I would preserve, is a KVT, because it's not so easy to change that object shape.

what is getRecipients(opts) ?

opts are the inputs given to sbot.db.create and include content, recps (alternative to content.recps), keys, feedFormat, encoding, etc. getRecipients takes an array of strings like sigil IDs and SSB URIs and returns an array of buffers.

I'm open to renaming this function such that we always reserve the word "recipient" for sigils and SSB URIs. But I don't what other word to use. Suggestions welcome. I surely want to avoid the generic word "key".

arj03 commented 2 years ago

This is quite a beast, but overall a really nice and needed refactor :-) I have gone through a first pass of the code now.

Some overall comments:

compat/publish.js

This is really nice!

db.js → core.js

Much better name, and really like how this is now just the core stuff.

addGroupKey

We should probably discuss how we define these group names/keys sooner rather than later.

staltz commented 2 years ago

db.js → core.js

@arj03 My idea with the rename is that you could use ssb-db2 without anything extra, i.e. .use(require('ssb-db2/core')) such that you add your own feed formats and plugins, etc. .use(require('ssb-db2/db')) wouldn't be as nice. I considered 'bare.js' as a name for a while, but went with core which is more common for us. If you do .use(require('ssb-db2')) then you're using reasonable defaults and built-in stuff, although opinionated.

arj03 commented 2 years ago

Yeah that is really nice. At some we can move out migrate as well, that will make package.json a lot leaner :)

staltz commented 2 years ago

well, it's an index of things you can decrypt. They are specifically not decrypted on disc, so I kind of liked that name better than decrypted ;)

I'm a tiny bit conflicted, and could move back to canDecrypt. But I'm going to try a bit more on this one: the index tells the records that we have successfully decrypted before, and the word is symmetric with "encrypted" in encrypted-box1.index.

But I'll give this some more time, let's see.

staltz commented 2 years ago

Beautiful :partying_face: https://github.com/ssbc/ssb-db2/pull/347/commits/f68c43bd51061d447a5f491cdbae14a2e6a3efe6

mixmix commented 2 years ago

@staltz re "getRecipients" do you mean it to be "getEnctptionKeys"?

You didn't say what the buffers were

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	855.78ms
Validate 5000 messages ssb-buttwoo	498.20ms
Validate 5000 messages sbot	465.91ms
Native to db format 5000 messages ssb-buttwoo	182.13ms
Native to db format 5000 messages sbot	121.44ms
Db to native format 5000 messages ssb-buttwoo	121.07ms
Db to native format 5000 messages sbot	129.73ms
Add 1000 elements	504.17ms
Add 1000 box1 msgs	1398.35ms
Unbox 1000 box1 msgs first run	145.56ms
Unbox 1000 box1 msgs second run	116.42ms
Add 1000 box1 msgs	1363.96ms
Query 1000 msgs first run	37.79ms
Query 1000 msgs second run	13.16ms
Add 1000 box2 msgs	1309.20ms
Unbox 1000 box2 msgs first run	105.74ms
Unbox 1000 box2 msgs second run	97.44ms
Migrate (+db1)	14873.16ms
Migrate (alone)	5735.57ms
Migrate (+db1 +db2)	11030.23ms
Migrate (+db2)	7752.45ms
Migrate continuation (+db2)	1258.02ms
Memory usage without indexes	746.66 MB = 33.23 MB + etc
Initial indexing	799.42ms
Initial indexing maxcpu=86	4695.45ms
Initial indexing compat	896.64ms
Two indexes updating concurrently	1073.72ms
Key one initial	62.93ms
Key two	2.03ms
Key one again	2.01ms
Reboot and key one again	56.12ms
Latest root posts	805.84ms
Latest posts	10.29ms
Votes one initial	649.79ms
Votes again	0.51ms
HasRoot	483.00ms
HasRoot again	0.25ms
Author one posts	401.53ms
Author two posts	18.39ms
Dedicated author one posts	399.81ms
Dedicated author one posts again	0.51ms
Maximum memory usage	1017.99 MB = 60.23 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	651.78ms
Validate 5000 messages ssb-buttwoo	459.39ms
Validate 5000 messages sbot	434.59ms
Native to db format 5000 messages ssb-buttwoo	154.70ms
Native to db format 5000 messages sbot	116.24ms
Db to native format 5000 messages ssb-buttwoo	109.94ms
Db to native format 5000 messages sbot	103.25ms
Add 1000 elements	488.96ms
Add 1000 box1 msgs	1191.66ms
Unbox 1000 box1 msgs first run	117.46ms
Unbox 1000 box1 msgs second run	81.81ms
Add 1000 box1 msgs	1162.63ms
Query 1000 msgs first run	33.12ms
Query 1000 msgs second run	12.49ms
Add 1000 box2 msgs	1115.14ms
Unbox 1000 box2 msgs first run	98.33ms
Unbox 1000 box2 msgs second run	87.17ms
Migrate (+db1)	11291.07ms
Migrate (alone)	3788.13ms
Migrate (+db1 +db2)	7835.04ms
Migrate (+db2)	5885.25ms
Migrate continuation (+db2)	855.68ms
Memory usage without indexes	751.02 MB = 39.33 MB + etc
Initial indexing	681.23ms
Initial indexing maxcpu=86	4081.75ms
Initial indexing compat	665.73ms
Two indexes updating concurrently	896.52ms
Key one initial	46.07ms
Key two	1.88ms
Key one again	0.80ms
Reboot and key one again	45.21ms
Latest root posts	658.89ms
Latest posts	8.80ms
Votes one initial	529.85ms
Votes again	0.42ms
HasRoot	381.06ms
HasRoot again	0.64ms
Author one posts	306.24ms
Author two posts	18.96ms
Dedicated author one posts	313.84ms
Dedicated author one posts again	0.50ms
Maximum memory usage	902.95 MB = 62.21 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	397.16ms
Validate 5000 messages	466.09ms
Native to db format 5000 messages	140.92ms
Db to native format 5000 messages	101.56ms
Add 1000 elements	497.23ms
Add 1000 box1 msgs	1228.86ms
Unbox 1000 box1 msgs first run	115.08ms
Unbox 1000 box1 msgs second run	93.53ms
Add 1000 box1 msgs	1191.40ms
Query 1000 msgs first run	38.07ms
Query 1000 msgs second run	13.13ms
Add 1000 box2 msgs	1161.24ms
Unbox 1000 box2 msgs first run	102.17ms
Unbox 1000 box2 msgs second run	82.45ms
Migrate (+db1)	10869.80ms
Migrate (alone)	3582.95ms
Migrate (+db1 +db2)	7737.19ms
Migrate (+db2)	5656.86ms
Migrate continuation (+db2)	879.07ms
Memory usage without indexes	741.22 MB = 41.29 MB + etc
Initial indexing	652.74ms
Initial indexing maxcpu=86	3887.11ms
Initial indexing compat	740.15ms
Two indexes updating concurrently	911.63ms
Key one initial	52.37ms
Key two	0.45ms
Key one again	0.57ms
Reboot and key one again	55.05ms
Latest root posts	566.60ms
Latest posts	8.63ms
Votes one initial	502.09ms
Votes again	0.60ms
HasRoot	363.52ms
HasRoot again	0.25ms
Author one posts	297.00ms
Author two posts	27.24ms
Dedicated author one posts	307.81ms
Dedicated author one posts again	0.43ms
Maximum memory usage	1004.38 MB = 65.37 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	423.48ms
Validate 5000 messages	461.64ms
Native to db format 5000 messages	147.63ms
Db to native format 5000 messages	118.68ms
Add 1000 elements	491.55ms
Add 1000 box1 msgs	929.22ms
Unbox 1000 box1 msgs first run	118.60ms
Unbox 1000 box1 msgs second run	84.93ms
Add 1000 box1 msgs	937.23ms
Query 1000 msgs first run	44.03ms
Query 1000 msgs second run	13.51ms
Add 1000 box2 msgs	1202.54ms
Unbox 1000 box2 msgs first run	91.40ms
Unbox 1000 box2 msgs second run	90.81ms
Migrate (+db1)	11815.20ms
Migrate (alone)	4015.82ms
Migrate (+db1 +db2)	8448.09ms
Migrate (+db2)	6140.11ms
Migrate continuation (+db2)	933.46ms
Memory usage without indexes	739.56 MB = 40.93 MB + etc
Initial indexing	705.95ms
Initial indexing maxcpu=86	4076.86ms
Initial indexing compat	696.14ms
Two indexes updating concurrently	908.15ms
Key one initial	43.34ms
Key two	0.54ms
Key one again	1.02ms
Reboot and key one again	42.59ms
Latest root posts	623.16ms
Latest posts	7.90ms
Votes one initial	552.46ms
Votes again	0.58ms
HasRoot	394.86ms
HasRoot again	0.29ms
Author one posts	300.37ms
Author two posts	15.83ms
Dedicated author one posts	334.45ms
Dedicated author one posts again	0.42ms
Maximum memory usage	1018.85 MB = 64.74 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	395.60ms
Validate 5000 messages	473.12ms
Native to db format 5000 messages	134.42ms
Db to native format 5000 messages	106.91ms
Add 1000 elements	493.14ms
Add 1000 box1 msgs	989.70ms
Unbox 1000 box1 msgs first run	116.28ms
Unbox 1000 box1 msgs second run	87.37ms
Add 1000 box1 msgs	991.63ms
Query 1000 msgs first run	38.08ms
Query 1000 msgs second run	12.95ms
Add 1000 box2 msgs	1213.17ms
Unbox 1000 box2 msgs first run	93.66ms
Unbox 1000 box2 msgs second run	88.99ms
Migrate (+db1)	10668.36ms
Migrate (alone)	3511.79ms
Migrate (+db1 +db2)	7689.78ms
Migrate (+db2)	5445.02ms
Migrate continuation (+db2)	835.51ms
Memory usage without indexes	739.50 MB = 39.94 MB + etc
Initial indexing	642.58ms
Initial indexing maxcpu=86	3756.50ms
Initial indexing compat	686.14ms
Two indexes updating concurrently	842.25ms
Key one initial	49.37ms
Key two	1.85ms
Key one again	0.77ms
Reboot and key one again	47.77ms
Latest root posts	543.70ms
Latest posts	8.85ms
Votes one initial	510.36ms
Votes again	1.12ms
HasRoot	342.07ms
HasRoot again	0.20ms
Author one posts	301.50ms
Author two posts	15.54ms
Dedicated author one posts	286.12ms
Dedicated author one posts again	0.45ms
Maximum memory usage	1012.27 MB = 62.49 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	356.65ms
Validate 5000 messages	402.11ms
Native to db format 5000 messages	137.43ms
Db to native format 5000 messages	98.23ms
Add 1000 elements	479.91ms
Add 1000 box1 msgs	834.32ms
Unbox 1000 box1 msgs first run	104.98ms
Unbox 1000 box1 msgs second run	87.19ms
Add 1000 box1 msgs	825.88ms
Query 1000 msgs first run	29.24ms
Query 1000 msgs second run	17.09ms
Add 1000 box2 msgs	1066.48ms
Unbox 1000 box2 msgs first run	90.44ms
Unbox 1000 box2 msgs second run	87.59ms
Migrate (+db1)	10953.63ms
Migrate (alone)	3703.00ms
Migrate (+db1 +db2)	7275.02ms
Migrate (+db2)	5642.29ms
Migrate continuation (+db2)	792.91ms
Memory usage without indexes	755.73 MB = 38.69 MB + etc
Initial indexing	599.70ms
Initial indexing maxcpu=86	3669.17ms
Initial indexing compat	557.47ms
Two indexes updating concurrently	865.39ms
Key one initial	45.30ms
Key two	0.58ms
Key one again	0.96ms
Reboot and key one again	42.95ms
Latest root posts	562.08ms
Latest posts	7.75ms
Votes one initial	488.02ms
Votes again	0.47ms
HasRoot	316.52ms
HasRoot again	0.26ms
Author one posts	303.48ms
Author two posts	16.89ms
Dedicated author one posts	303.79ms
Dedicated author one posts again	0.41ms
Maximum memory usage	1037.48 MB = 68.56 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	413.52ms
Validate 5000 messages	452.61ms
Native to db format 5000 messages	141.63ms
Db to native format 5000 messages	115.31ms
Add 1000 elements	489.43ms
Add 1000 box1 msgs	934.19ms
Unbox 1000 box1 msgs first run	110.03ms
Unbox 1000 box1 msgs second run	85.44ms
Add 1000 box1 msgs	928.65ms
Query 1000 msgs first run	43.69ms
Query 1000 msgs second run	14.16ms
Add 1000 box2 msgs	1183.36ms
Unbox 1000 box2 msgs first run	91.56ms
Unbox 1000 box2 msgs second run	87.65ms
Migrate (+db1)	11316.35ms
Migrate (alone)	3966.55ms
Migrate (+db1 +db2)	8070.00ms
Migrate (+db2)	5980.01ms
Migrate continuation (+db2)	845.19ms
Memory usage without indexes	743.95 MB = 40.70 MB + etc
Initial indexing	684.87ms
Initial indexing maxcpu=86	4087.15ms
Initial indexing compat	665.66ms
Two indexes updating concurrently	939.37ms
Key one initial	52.69ms
Key two	0.48ms
Key one again	1.64ms
Reboot and key one again	43.07ms
Latest root posts	627.13ms
Latest posts	9.52ms
Votes one initial	534.75ms
Votes again	0.58ms
HasRoot	399.82ms
HasRoot again	0.23ms
Author one posts	325.94ms
Author two posts	18.82ms
Dedicated author one posts	306.93ms
Dedicated author one posts again	0.53ms
Maximum memory usage	1012.38 MB = 63.11 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	399.95ms
Validate 5000 messages	474.25ms
Native to db format 5000 messages	140.93ms
Db to native format 5000 messages	103.86ms
Add 1000 elements	492.92ms
Add 1000 box1 msgs	999.51ms
Unbox 1000 box1 msgs first run	118.46ms
Unbox 1000 box1 msgs second run	98.54ms
Add 1000 box1 msgs	992.93ms
Query 1000 msgs first run	31.72ms
Query 1000 msgs second run	12.15ms
Add 1000 box2 msgs	1208.97ms
Unbox 1000 box2 msgs first run	107.39ms
Unbox 1000 box2 msgs second run	83.99ms
Migrate (+db1)	10702.14ms
Migrate (alone)	3448.20ms
Migrate (+db1 +db2)	7846.56ms
Migrate (+db2)	5519.35ms
Migrate continuation (+db2)	780.82ms
Memory usage without indexes	736.45 MB = 38.40 MB + etc
Initial indexing	624.59ms
Initial indexing maxcpu=86	3867.68ms
Initial indexing compat	612.21ms
Two indexes updating concurrently	857.87ms
Key one initial	46.79ms
Key two	0.47ms
Key one again	1.05ms
Reboot and key one again	48.66ms
Latest root posts	524.71ms
Latest posts	17.12ms
Votes one initial	498.10ms
Votes again	0.95ms
HasRoot	371.58ms
HasRoot again	0.35ms
Author one posts	319.09ms
Author two posts	16.07ms
Dedicated author one posts	314.83ms
Dedicated author one posts again	0.44ms
Maximum memory usage	1033.91 MB = 64.50 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	405.09ms
Validate 5000 messages	447.08ms
Native to db format 5000 messages	161.12ms
Db to native format 5000 messages	113.86ms
Add 1000 elements	497.22ms
Add 1000 box1 msgs	941.81ms
Unbox 1000 box1 msgs first run	114.18ms
Unbox 1000 box1 msgs second run	98.09ms
Add 1000 box1 msgs	939.21ms
Query 1000 msgs first run	34.57ms
Query 1000 msgs second run	21.32ms
Add 1000 box2 msgs	1202.93ms
Unbox 1000 box2 msgs first run	92.88ms
Unbox 1000 box2 msgs second run	92.58ms
Migrate (+db1)	12475.29ms
Migrate (alone)	4190.11ms
Migrate (+db1 +db2)	8353.78ms
Migrate (+db2)	6468.94ms
Migrate continuation (+db2)	1083.40ms
Memory usage without indexes	745.36 MB = 38.32 MB + etc
Initial indexing	708.56ms
Initial indexing maxcpu=86	4089.90ms
Initial indexing compat	673.42ms
Two indexes updating concurrently	978.41ms
Key one initial	50.49ms
Key two	2.37ms
Key one again	0.75ms
Reboot and key one again	46.51ms
Latest root posts	605.14ms
Latest posts	10.49ms
Votes one initial	533.35ms
Votes again	0.60ms
HasRoot	366.71ms
HasRoot again	0.46ms
Author one posts	334.31ms
Author two posts	15.98ms
Dedicated author one posts	337.08ms
Dedicated author one posts again	0.31ms
Maximum memory usage	1028.44 MB = 71.32 MB + etc
Indexes folder size	10.12mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	420.26ms
Validate 5000 messages	443.60ms
Native to db format 5000 messages	157.95ms
Db to native format 5000 messages	112.21ms
Add 1000 elements	528.14ms
Add 1000 box1 msgs	931.75ms
Unbox 1000 box1 msgs first run	115.62ms
Unbox 1000 box1 msgs second run	85.26ms
Add 1000 box1 msgs	919.75ms
Query 1000 msgs first run	35.03ms
Query 1000 msgs second run	14.63ms
Add 1000 box2 msgs	1196.30ms
Unbox 1000 box2 msgs first run	107.07ms
Unbox 1000 box2 msgs second run	80.72ms
Migrate (+db1)	11403.48ms
Migrate (alone)	3928.55ms
Migrate (+db1 +db2)	8212.36ms
Migrate (+db2)	6117.45ms
Migrate continuation (+db2)	880.68ms
Memory usage without indexes	747.58 MB = 38.70 MB + etc
Initial indexing	711.87ms
Initial indexing maxcpu=86	3934.93ms
Initial indexing compat	738.18ms
Two indexes updating concurrently	1014.72ms
Key one initial	44.73ms
Key two	0.50ms
Key one again	0.80ms
Reboot and key one again	42.51ms
Latest root posts	624.28ms
Latest posts	8.73ms
Votes one initial	557.23ms
Votes again	0.62ms
HasRoot	353.38ms
HasRoot again	0.63ms
Author one posts	339.62ms
Author two posts	15.73ms
Dedicated author one posts	339.71ms
Dedicated author one posts again	0.52ms
Maximum memory usage	1004.09 MB = 65.96 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	414.06ms
Validate 5000 messages	451.26ms
Native to db format 5000 messages	139.54ms
Db to native format 5000 messages	111.10ms
Add 1000 elements	491.47ms
Add 1000 box1 msgs	922.70ms
Unbox 1000 box1 msgs first run	116.55ms
Unbox 1000 box1 msgs second run	84.03ms
Add 1000 box1 msgs	929.37ms
Query 1000 msgs first run	44.90ms
Query 1000 msgs second run	13.80ms
Add 1000 box2 msgs	1206.79ms
Unbox 1000 box2 msgs first run	118.42ms
Unbox 1000 box2 msgs second run	104.41ms
Migrate (+db1)	11525.02ms
Migrate (alone)	3817.46ms
Migrate (+db1 +db2)	8200.92ms
Migrate (+db2)	6081.30ms
Migrate continuation (+db2)	889.30ms
Memory usage without indexes	756.30 MB = 40.07 MB + etc
Initial indexing	669.26ms
Initial indexing maxcpu=86	4013.10ms
Initial indexing compat	611.24ms
Two indexes updating concurrently	1014.39ms
Key one initial	48.93ms
Key two	1.92ms
Key one again	0.81ms
Reboot and key one again	44.13ms
Latest root posts	579.26ms
Latest posts	9.17ms
Votes one initial	542.64ms
Votes again	0.64ms
HasRoot	352.65ms
HasRoot again	0.72ms
Author one posts	321.65ms
Author two posts	15.77ms
Dedicated author one posts	324.56ms
Dedicated author one posts again	0.27ms
Maximum memory usage	1008.96 MB = 65.83 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	534.42ms
Validate 5000 messages	523.98ms
Native to db format 5000 messages	174.48ms
Db to native format 5000 messages	138.19ms
Add 1000 elements	518.35ms
Add 1000 box1 msgs	1172.79ms
Unbox 1000 box1 msgs first run	161.59ms
Unbox 1000 box1 msgs second run	124.84ms
Add 1000 box1 msgs	1165.07ms
Query 1000 msgs first run	39.44ms
Query 1000 msgs second run	21.59ms
Add 1000 box2 msgs	1510.28ms
Unbox 1000 box2 msgs first run	126.83ms
Unbox 1000 box2 msgs second run	99.67ms
Migrate (+db1)	15084.20ms
Migrate (alone)	5753.60ms
Migrate (+db1 +db2)	11055.38ms
Migrate (+db2)	7966.83ms
Migrate continuation (+db2)	1199.30ms
Memory usage without indexes	744.31 MB = 39.15 MB + etc
Initial indexing	910.61ms
Initial indexing maxcpu=86	4869.54ms
Initial indexing compat	993.89ms
Two indexes updating concurrently	1144.55ms
Key one initial	55.55ms
Key two	0.49ms
Key one again	0.80ms
Reboot and key one again	52.35ms
Latest root posts	786.47ms
Latest posts	12.67ms
Votes one initial	660.82ms
Votes again	1.11ms
HasRoot	428.03ms
HasRoot again	0.23ms
Author one posts	444.18ms
Author two posts	17.38ms
Dedicated author one posts	365.84ms
Dedicated author one posts again	0.72ms
Maximum memory usage	1010.46 MB = 63.40 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	400.71ms
Validate 5000 messages	446.11ms
Native to db format 5000 messages	155.05ms
Db to native format 5000 messages	107.23ms
Add 1000 elements	493.02ms
Add 1000 box1 msgs	936.15ms
Unbox 1000 box1 msgs first run	116.52ms
Unbox 1000 box1 msgs second run	90.69ms
Add 1000 box1 msgs	928.40ms
Query 1000 msgs first run	31.66ms
Query 1000 msgs second run	20.01ms
Add 1000 box2 msgs	1191.40ms
Unbox 1000 box2 msgs first run	100.47ms
Unbox 1000 box2 msgs second run	96.29ms
Migrate (+db1)	11334.32ms
Migrate (alone)	3833.12ms
Migrate (+db1 +db2)	7997.74ms
Migrate (+db2)	6065.56ms
Migrate continuation (+db2)	930.94ms
Memory usage without indexes	761.00 MB = 43.25 MB + etc
Initial indexing	712.67ms
Initial indexing maxcpu=86	4170.30ms
Initial indexing compat	725.08ms
Two indexes updating concurrently	904.28ms
Key one initial	41.41ms
Key two	0.61ms
Key one again	1.00ms
Reboot and key one again	48.27ms
Latest root posts	587.38ms
Latest posts	9.46ms
Votes one initial	545.82ms
Votes again	0.49ms
HasRoot	392.66ms
HasRoot again	0.70ms
Author one posts	340.71ms
Author two posts	13.90ms
Dedicated author one posts	306.18ms
Dedicated author one posts again	0.25ms
Maximum memory usage	917.58 MB = 61.51 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	464.45ms
Validate 5000 messages	536.70ms
Native to db format 5000 messages	163.90ms
Db to native format 5000 messages	131.38ms
Add 1000 elements	510.10ms
Add 1000 box1 msgs	1102.61ms
Unbox 1000 box1 msgs first run	148.17ms
Unbox 1000 box1 msgs second run	98.59ms
Add 1000 box1 msgs	1110.42ms
Query 1000 msgs first run	56.78ms
Query 1000 msgs second run	16.19ms
Add 1000 box2 msgs	1413.40ms
Unbox 1000 box2 msgs first run	115.41ms
Unbox 1000 box2 msgs second run	99.45ms
Migrate (+db1)	13722.77ms
Migrate (alone)	4773.51ms
Migrate (+db1 +db2)	10164.64ms
Migrate (+db2)	7523.09ms
Migrate continuation (+db2)	1103.90ms
Memory usage without indexes	747.22 MB = 39.75 MB + etc
Initial indexing	831.33ms
Initial indexing maxcpu=86	4730.98ms
Initial indexing compat	967.63ms
Two indexes updating concurrently	1159.03ms
Key one initial	58.16ms
Key two	0.75ms
Key one again	0.33ms
Reboot and key one again	51.19ms
Latest root posts	930.59ms
Latest posts	11.57ms
Votes one initial	696.97ms
Votes again	0.58ms
HasRoot	477.67ms
HasRoot again	0.41ms
Author one posts	612.22ms
Author two posts	18.52ms
Dedicated author one posts	566.09ms
Dedicated author one posts again	0.48ms
Maximum memory usage	1037.09 MB = 59.72 MB + etc
Indexes folder size	10.12mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	394.02ms
Validate 5000 messages	474.20ms
Native to db format 5000 messages	134.59ms
Db to native format 5000 messages	105.78ms
Add 1000 elements	494.26ms
Add 1000 box1 msgs	1001.26ms
Unbox 1000 box1 msgs first run	121.74ms
Unbox 1000 box1 msgs second run	91.13ms
Add 1000 box1 msgs	995.97ms
Query 1000 msgs first run	39.72ms
Query 1000 msgs second run	13.14ms
Add 1000 box2 msgs	1199.57ms
Unbox 1000 box2 msgs first run	96.68ms
Unbox 1000 box2 msgs second run	90.89ms
Migrate (+db1)	10532.55ms
Migrate (alone)	3494.79ms
Migrate (+db1 +db2)	8000.63ms
Migrate (+db2)	5494.65ms
Migrate continuation (+db2)	854.63ms
Memory usage without indexes	737.48 MB = 39.12 MB + etc
Initial indexing	647.80ms
Initial indexing maxcpu=86	3851.42ms
Initial indexing compat	691.75ms
Two indexes updating concurrently	910.19ms
Key one initial	49.33ms
Key two	0.93ms
Key one again	0.80ms
Reboot and key one again	41.69ms
Latest root posts	700.77ms
Latest posts	9.01ms
Votes one initial	542.46ms
Votes again	1.04ms
HasRoot	345.33ms
HasRoot again	0.31ms
Author one posts	447.72ms
Author two posts	15.20ms
Dedicated author one posts	454.10ms
Dedicated author one posts again	0.44ms
Maximum memory usage	1005.38 MB = 62.29 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	461.02ms
Validate 5000 messages	513.21ms
Native to db format 5000 messages	167.74ms
Db to native format 5000 messages	132.68ms
Add 1000 elements	510.80ms
Add 1000 box1 msgs	1077.46ms
Unbox 1000 box1 msgs first run	144.36ms
Unbox 1000 box1 msgs second run	91.83ms
Add 1000 box1 msgs	1038.55ms
Query 1000 msgs first run	47.08ms
Query 1000 msgs second run	15.82ms
Add 1000 box2 msgs	407.87ms
Unbox 1000 box2 msgs first run	206.77ms
Unbox 1000 box2 msgs second run	192.21ms
Add 1000 box2 group msgs	271.73ms
Unbox 1000 box2 group msgs first run	3556.37ms
Unbox 1000 box2 group msgs second run	2989.77ms
Migrate (+db1)	13576.49ms
Migrate (alone)	4692.88ms
Migrate (+db1 +db2)	10055.30ms
Migrate (+db2)	7096.01ms
Migrate continuation (+db2)	1048.92ms
Memory usage without indexes	758.05 MB = 38.46 MB + etc
Initial indexing	775.23ms
Initial indexing maxcpu=86	5007.91ms
Initial indexing compat	913.82ms
Two indexes updating concurrently	1127.74ms
Key one initial	56.10ms
Key two	2.07ms
Key one again	2.20ms
Reboot and key one again	62.83ms
Latest root posts	876.93ms
Latest posts	9.29ms
Votes one initial	612.81ms
Votes again	0.46ms
HasRoot	401.35ms
HasRoot again	0.51ms
Author one posts	538.29ms
Author two posts	18.30ms
Dedicated author one posts	565.11ms
Dedicated author one posts again	0.40ms
Maximum memory usage	1024.83 MB = 65.42 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	389.21ms
Validate 5000 messages	452.34ms
Native to db format 5000 messages	149.31ms
Db to native format 5000 messages	116.03ms
Add 1000 elements	490.78ms
Add 1000 box1 msgs	939.64ms
Unbox 1000 box1 msgs first run	125.22ms
Unbox 1000 box1 msgs second run	84.28ms
Add 1000 box1 msgs	903.44ms
Query 1000 msgs first run	38.60ms
Query 1000 msgs second run	13.43ms
Add 1000 box2 msgs	381.50ms
Unbox 1000 box2 msgs first run	176.99ms
Unbox 1000 box2 msgs second run	163.24ms
Add 1000 box2 group msgs	243.24ms
Unbox 1000 box2 group msgs first run	3048.21ms
Unbox 1000 box2 group msgs second run	2612.44ms
Migrate (+db1)	11367.09ms
Migrate (alone)	4011.80ms
Migrate (+db1 +db2)	8284.15ms
Migrate (+db2)	6273.20ms
Migrate continuation (+db2)	987.36ms
Memory usage without indexes	754.80 MB = 41.81 MB + etc
Initial indexing	725.76ms
Initial indexing maxcpu=86	4011.21ms
Initial indexing compat	743.56ms
Two indexes updating concurrently	1008.82ms
Key one initial	45.44ms
Key two	0.51ms
Key one again	0.57ms
Reboot and key one again	53.43ms
Latest root posts	710.22ms
Latest posts	8.34ms
Votes one initial	543.32ms
Votes again	0.47ms
HasRoot	380.66ms
HasRoot again	0.44ms
Author one posts	481.89ms
Author two posts	17.03ms
Dedicated author one posts	465.79ms
Dedicated author one posts again	0.67ms
Maximum memory usage	1027.81 MB = 63.34 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	415.18ms
Validate 5000 messages	483.99ms
Native to db format 5000 messages	137.80ms
Db to native format 5000 messages	109.31ms
Add 1000 elements	496.04ms
Add 1000 box1 msgs	1007.05ms
Unbox 1000 box1 msgs first run	168.35ms
Unbox 1000 box1 msgs second run	102.95ms
Add 1000 box1 msgs	967.81ms
Query 1000 msgs first run	46.09ms
Query 1000 msgs second run	23.49ms
Add 1000 box2 msgs	359.01ms
Unbox 1000 box2 msgs first run	223.61ms
Unbox 1000 box2 msgs second run	157.51ms
Add 1000 box2 group msgs	233.92ms
Unbox 1000 box2 group msgs first run	369.83ms
Unbox 1000 box2 group msgs second run	316.64ms
Migrate (+db1)	11568.91ms
Migrate (alone)	3715.99ms
Migrate (+db1 +db2)	8559.33ms
Migrate (+db2)	5951.80ms
Migrate continuation (+db2)	904.12ms
Memory usage without indexes	757.24 MB = 38.52 MB + etc
Initial indexing	657.39ms
Initial indexing maxcpu=86	4082.20ms
Initial indexing compat	750.10ms
Two indexes updating concurrently	959.33ms
Key one initial	52.80ms
Key two	1.13ms
Key one again	1.03ms
Reboot and key one again	59.69ms
Latest root posts	747.54ms
Latest posts	9.49ms
Votes one initial	530.20ms
Votes again	0.52ms
HasRoot	360.79ms
HasRoot again	0.30ms
Author one posts	446.18ms
Author two posts	23.01ms
Dedicated author one posts	470.78ms
Dedicated author one posts again	0.49ms
Maximum memory usage	1018.86 MB = 63.27 MB + etc
Indexes folder size	10.26mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	400.67ms
Validate 5000 messages	473.65ms
Native to db format 5000 messages	134.81ms
Db to native format 5000 messages	111.13ms
Add 1000 elements	493.27ms
Add 1000 box1 msgs	994.75ms
Unbox 1000 box1 msgs first run	165.06ms
Unbox 1000 box1 msgs second run	107.50ms
Add 1000 box1 msgs	959.07ms
Query 1000 msgs first run	38.12ms
Query 1000 msgs second run	20.26ms
Add 1000 box2 msgs	342.28ms
Unbox 1000 box2 msgs first run	192.73ms
Unbox 1000 box2 msgs second run	147.93ms
Add 1000 box2 group msgs	222.45ms
Unbox 1000 box2 group msgs first run	340.92ms
Unbox 1000 box2 group msgs second run	267.76ms
Migrate (+db1)	10601.83ms
Migrate (alone)	3577.49ms
Migrate (+db1 +db2)	7835.60ms
Migrate (+db2)	5446.25ms
Migrate continuation (+db2)	828.40ms
Memory usage without indexes	736.18 MB = 38.59 MB + etc
Initial indexing	631.03ms
Initial indexing maxcpu=86	3874.24ms
Initial indexing compat	572.47ms
Two indexes updating concurrently	926.79ms
Key one initial	56.05ms
Key two	1.02ms
Key one again	0.82ms
Reboot and key one again	50.39ms
Latest root posts	709.31ms
Latest posts	10.21ms
Votes one initial	512.19ms
Votes again	0.37ms
HasRoot	351.62ms
HasRoot again	0.28ms
Author one posts	445.27ms
Author two posts	15.19ms
Dedicated author one posts	446.69ms
Dedicated author one posts again	0.49ms
Maximum memory usage	1011.68 MB = 63.59 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	588.05ms
Validate 5000 messages	587.10ms
Native to db format 5000 messages	184.30ms
Db to native format 5000 messages	149.30ms
Add 1000 elements	526.45ms
Add 1000 box1 msgs	1264.98ms
Unbox 1000 box1 msgs first run	214.58ms
Unbox 1000 box1 msgs second run	143.74ms
Add 1000 box1 msgs	1223.85ms
Query 1000 msgs first run	50.33ms
Query 1000 msgs second run	15.71ms
Add 1000 box2 msgs	518.08ms
Unbox 1000 box2 msgs first run	286.39ms
Unbox 1000 box2 msgs second run	199.56ms
Add 1000 box2 group msgs	347.82ms
Unbox 1000 box2 group msgs first run	467.57ms
Unbox 1000 box2 group msgs second run	350.46ms
Migrate (+db1)	15267.02ms
Migrate (alone)	5388.82ms
Migrate (+db1 +db2)	11107.82ms
Migrate (+db2)	8166.71ms
Migrate continuation (+db2)	1364.92ms
Memory usage without indexes	756.90 MB = 38.34 MB + etc
Initial indexing	1023.44ms
Initial indexing maxcpu=86	4681.62ms
Initial indexing compat	981.36ms
Two indexes updating concurrently	1281.37ms
Key one initial	66.45ms
Key two	1.02ms
Key one again	1.06ms
Reboot and key one again	59.98ms
Latest root posts	975.10ms
Latest posts	11.27ms
Votes one initial	668.13ms
Votes again	0.76ms
HasRoot	471.42ms
HasRoot again	0.43ms
Author one posts	594.95ms
Author two posts	21.32ms
Dedicated author one posts	633.38ms
Dedicated author one posts again	0.62ms
Maximum memory usage	1035.14 MB = 60.38 MB + etc
Indexes folder size	10.01mb

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	359.69ms
Validate 5000 messages	455.61ms
Native to db format 5000 messages	114.45ms
Db to native format 5000 messages	83.55ms
Add 1000 elements	490.98ms
Add 1000 box1 msgs	940.63ms
Unbox 1000 box1 msgs first run	160.11ms
Unbox 1000 box1 msgs second run	107.36ms
Add 1000 box1 msgs	803.85ms
Query 1000 msgs first run	34.42ms
Query 1000 msgs second run	23.32ms
Add 1000 box2 msgs	1360.94ms
Unbox 1000 box2 msgs first run	228.75ms
Unbox 1000 box2 msgs second run	166.74ms
Add 1000 box2 group msgs	1225.32ms
Unbox 1000 box2 group msgs first run	370.94ms
Unbox 1000 box2 group msgs second run	303.18ms
Migrate (+db1)	11560.47ms
Migrate (alone)	4011.39ms
Migrate (+db1 +db2)	7424.37ms
Migrate (+db2)	6349.99ms
Migrate continuation (+db2)	1753.78ms
Memory usage without indexes	757.22 MB = 36.42 MB + etc
Initial indexing	1521.48ms
Initial indexing maxcpu=86	4079.93ms
Initial indexing compat	1524.42ms
Two indexes updating concurrently	1799.47ms
Key one initial	55.92ms
Key two	2.09ms
Key one again	1.72ms
Reboot and key one again	46.66ms
Latest root posts	1930.01ms
Latest posts	7.39ms
Votes one initial	486.15ms
Votes again	0.36ms
HasRoot	371.16ms
HasRoot again	1.26ms
Author one posts	483.37ms
Author two posts	33.82ms
Dedicated author one posts	471.81ms
Dedicated author one posts again	0.45ms
Maximum memory usage	933.52 MB = 67.79 MB + etc
Indexes folder size	10.01mb

staltz commented 2 years ago

@arj03 I replaced encryption-formats/box2 with ssb-box2 module and tests failed for a reason that is very anticlimax:

Uncaught Exception at  uncaughtException : Error [OpenError]: 
IO error: lock /tmp/ssb-db2-about-self-index/keyring/LOCK: 
already held by process

This means that the level instance in ssb-keyring needs to be closed when the sbot is closed. =P

This requires some boring plumbing:

encryptionFormat "spec" can have teardown method
ssb-keyring needs to be updated with a close API
ssb-box2 needs to have a teardown method that calls ssb-keyring close
ssb-db2 needs to call teardown for all encryption formats, when sbot is closing

To be done.

staltz commented 2 years ago

Okay, this branch is ALMOST done. encryption-formats/box2 was replaced with ssb-box2 module, and encryption formats teardown() is called on sbot.close. All tests pass except a simple one, pending on https://github.com/ssbc/ssb-private-group-keys/pull/14 being fixed and published. Before we publish the next major version of ssb-db2 with formats, it would be good that ssb-keyring depends on a published version of ssb-private-group-keys not on the PR branch.

staltz commented 2 years ago

I'm gonna fix the benchmarks before this is ready for review.

arj03 commented 2 years ago

Some review notes:

the opts that add, addOOO, addOOOBatch, addBatch, addTransaction now can take needs to be documented
Can we make add() and friends default to encoding = js like create does?

Do you remember why we pin ssb-friends to 5.1.0?

arj03 commented 2 years ago

Looking at create and the encryptionFormat option. If you don't specify encryptionFormat, the function will just try each format to find one that works. Maybe we should default to box instead?

arj03 commented 2 years ago

Overall I think this looks really good. I love how core.js is now really streamlined. It's hard to review such a big PR, not really a fault, just mentioning it to say that we might find bugs when we start upgrading other modules to work with this. I'm happy that we have a relatively good test-suite (and benchmarks :)). So in order to get move things along and really test I'll say it's fine to merge this in once the minor things I mentioned are fixed together with the benchmarks as you mentioned. It is after all a major version change.

staltz commented 2 years ago

Thanks @arj03 for taking a look, I'm working on those.

Two notes:

I disabled the benchmark that uses box2 groups simply because we still don't have the removeGroupKeys API in ssb-box2 :(
In index.js, I put both ssb-box and ssb-box2, do you think it that ssb-box2 should not included (and the user has to install the plugin themselves)? Previously, ssb-db2 didn't support box2 out of the box, you had to install ssb-db2-box2

arj03 commented 2 years ago

I think it's good to include both box1 and 2 in index.js. just that the default would be box1 of you don't specify in create. It is still the case that most clients can't decide box2.

staltz commented 2 years ago

the opts that add, addOOO, addOOOBatch, addBatch, addTransaction now can take needs to be documented

Yes, done

Can we make add() and friends default to encoding = js like create does?

Yes, done

Do you remember why we pin ssb-friends to 5.1.0?

Oh, that's just a devDep and it's only used for the benchmarks (and actually those specific benchmarks are skipped at the moment). It's pinned so that the benchmarks are always using the same ssb-friends and we would just be testing changes in ssb-db2 (not, for instance, performance improvements in ssb-friends).

staltz commented 2 years ago

Looking at create and the encryptionFormat option. If you don't specify encryptionFormat, the function will just try each format to find one that works. Maybe we should default to box instead?

This actually made a lot of sense, and I made the change. But then some tests broke. Specifically, if you use ssb.db.publish() with some private group as a recp, you don't have any way of specifying which encryptionFormat to use, so it'll use box and that's going to fail.

I'm inclined to use ssb.db.create() in those tests, but I wanted to let you know that if we do the change you suggested, you won't be able to use box2 with ssb.db.publish().

arj03 commented 2 years ago

I think that is for the best actually as publish is legacy anyway

staltz commented 2 years ago

Yeah, I'm a bit conflicted.

On one hand, removing the "try encryption formats until one of them works" is inconsistent with the default setting for encoding and feed format.

On the other hand, now with create() you have to always specify encryptionFormat: 'box2' even if it's implicit by the recp being a group ID.

And on the third (??) hand, since encryption is important and you don't want to screw it up, it's important to explicitly mention box2 not leaving it implicit.

staltz commented 2 years ago

Okay but I implemented it anyway. Ready for Final FINAL review.

arj03 commented 2 years ago

I think it will be for the best this way :) Nice to see the last 2 commits actually removed a bunch of lines

arj03 commented 2 years ago

Can you create an issue for the benchmark that was commented out?

github-actions[bot] commented 2 years ago

Benchmark results

Part	Duration
Create 5000 new messages	464.53ms
Validate 5000 messages	582.99ms
Native to db format 5000 messages	142.79ms
Db to native format 5000 messages	107.26ms
Add 1000 elements	524.56ms
Add 1000 box1 msgs	1213.88ms
Unbox 1000 box1 msgs first run	211.74ms
Unbox 1000 box1 msgs second run	145.35ms
Add 1000 box1 msgs	1205.58ms
Query 1000 msgs first run	56.40ms
Query 1000 msgs second run	17.34ms
Add 1000 box2 msgs	1634.10ms
Unbox 1000 box2 msgs first run	250.68ms
Unbox 1000 box2 msgs second run	177.72ms
Migrate (+db1)	14861.15ms
Migrate (alone)	5139.81ms
Migrate (+db1 +db2)	11518.36ms
Migrate (+db2)	7670.72ms
Migrate continuation (+db2)	1300.02ms
Memory usage without indexes	775.54 MB = 37.50 MB + etc
Initial indexing	877.53ms
Initial indexing maxcpu=86	4695.34ms
Initial indexing compat	997.14ms
Two indexes updating concurrently	1389.13ms
Key one initial	61.60ms
Key two	1.94ms
Key one again	1.94ms
Reboot and key one again	64.26ms
Latest root posts	1018.32ms
Latest posts	17.41ms
Votes one initial	736.68ms
Votes again	1.52ms
HasRoot	482.18ms
HasRoot again	0.55ms
Author one posts	589.18ms
Author two posts	23.76ms
Dedicated author one posts	653.76ms
Dedicated author one posts again	0.39ms
DeleteFeed	3647.04ms
Maximum memory usage	1001.49 MB = 64.49 MB + etc
Indexes folder size	10.01mb

staltz commented 2 years ago

MERGEDDDDDDD

ssbc / ssb-db2

feedFormats and encryptionFormats as plugins #347

Context

Solution

:large_blue_circle: Encoding

:yellow_circle: Feed Format

:black_circle: Encryption Format

TODO

db => network

network => db

:horse_racing: :lizard: :melon:

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results

Benchmark results