feedFormats and encryptionFormats as plugins

[staltz]

Context

We are adding buttwoo as a feed format, and it works differently than other formats, so we would introduce a lot of complexity. At the same time, we need to improve the situation of support for box2 before we move on with "ssb-tribes2" on ssb-db2.

Solution

sbot.db.installFeedFormat() and sbot.db.installEncryptionFormat() and encodings.

These are now orthogonal concerns, so any encryption format can work with any feed format, which can support different encodings.

:large_blue_circle: Encoding

An "encoding" is a way of representing a feed message. This PR supports two encodings: 'js' (JavaScript objects) and 'bipf' (BIPF buffers).

:yellow_circle: Feed Format

A feed format defines how to create messages that follow a schema and a spec. Every feed format is a plugin-like object with:

• name string
• encodings array of supported encoding names
• functions to create and convert "native messages"
  • newNativeMsg(opts)
  • toNativeMsg(msgVal, encoding)
  • fromNativeMsg(nativeMsg, encoding)
  • fromDecryptedNativeMsg(plaintextBuf, nativeMsg, encoding)
• encryption-related function
  • toPlaintextBuffer(opts)
• helper functions getX() and isX()
  • getFeedId(nativeMsg)
  • getMsgId(nativeMsg)
  • isNativeMsg(x)
  • isAuthor(author)
• validation functions
  • validateSingle()
  • validateBatch()
  • validateOOOBatch()

:star: "Native Msg" this concept is new and important. Every feed format has a "native message format", which is a shape for the message that is specific to the feed format. The feed format fully controls its own "nativeMsg" shape, and it can be anything.

• For "classic", a nativeMsg is the traditional JavaScript object msgVal.
• For "bendybutt-v1", a nativeMsg is a BFE-and-bencoded buffer with a shape determined by bendy-butt-spec.
• For "buttwoo-v1", a nativeMsg is a BFE-and-bipf buffer with a shape determined by buttwoo-spec.

:black_circle: Encryption Format

An encryption format specifies how to encrypt and decrypt JavaScript buffers intended for some specific recipients. Every encryption format is a plugin-like object with:

• name string, also used for the suffix on the ciphertext
• setup(config, cb) OPTIONAL
• encrypt(plaintextBuf, opts) => ciphertextBuf
• decrypt(ciphertextBuf, opts) => plaintextBuf

TODO

• [x] private.js special treatments of box1 and box2
• [x] Implement publish in terms of create()
• [x] isPrivate(encryptionFormat) for querying canDecrypt
• [x] Move post Obv to compat
• [x] :wrench: feedFormat.getSequence() (needed for ebt)
• [x] :detective: Is buttwoo performant with ssb-ebt?
• [x] :detective: WeakMap optimizations in feed formats, from/to native msg
• [x] :detective: Figure out box2 key management/addition
• [x] :detective: jitdb operator for querying encrypted-box1 and encrypted-box2
• [x] :wrench: Restore reindex-encrypted.js tests
• [x] :wrench: ssb-db2 should wait for all encryption formats to load
• [x] :wrench: Fix benchmarks
• [x] :abc: Better name for encryptionFormat.getRecipients() => getEncryptionKeys()
• [x] :abc: canDecrypt.index naming?
• [x] :balance_scale: ssb-keyring LGPL-3.0 release
• [x] :rocket: Publish ssb-feed-format
• [x] :rocket: Publish ssb-encryption-format
• [x] :rocket: Publish ssb-classic
• [x] :rocket: Breaking change ssb-bendy-butt
• [x] :rocket: Publish ssb-box1
• [x] :wrench: ssb-uri-spec buttwoo URI needs parent
• [x] :wrench: ssb-uri2 buttwoo URI needs parent
• [x] :rocket: Breaking change ssb-buttwoo
• [x] :book: Document new APIs such as:
  • create()
  • onMsgAdded()
  • installFeedFormat()
  • installEncryptionFormat()
• [x] :book: Document breaking changes such as:
  • encrypted.index/canDecrypt.index
  • setPost hack for partial replication in the browser
  • isPrivate() => isDecrypted() / isEncrypted()
• [x] :wrench: ssb-keyring changes to support derived-keys-only workflow
• [x] :rocket: Publish ssb-box2
• [x] MERGE AND PUBLISH VERSION 5.0.0
• [x] ssb-ebt Replace post with onMsgAdded
• [x] ssb-conn Replace post with onMsgAdded
• [x] ssb-meta-feeds Update ssb-bendy-butt to new feed format API
• [x] ssb-subset-ql Update with new db2 5.0.0

[github-actions[bot]]

Benchmark results

Part	Duration
Create 5000 new messages	455.87ms
Validate 5000 messages	549.00ms
Native to db format 5000 messages	148.89ms
Db to native format 5000 messages	117.29ms
Add 1000 elements	514.47ms
Add 1000 box1 msgs	1193.43ms
Unbox 1000 box1 msgs first run	215.57ms
Unbox 1000 box1 msgs second run	149.08ms
Add 1000 box1 msgs	1143.64ms
Query 1000 msgs first run	47.76ms
Query 1000 msgs second run	31.01ms
Add 1000 box2 msgs	1572.71ms
Unbox 1000 box2 msgs first run	270.02ms
Unbox 1000 box2 msgs second run	189.57ms
Migrate (+db1)	14391.50ms
Migrate (alone)	4862.54ms
Migrate (+db1 +db2)	10710.75ms
Migrate (+db2)	7694.13ms
Migrate continuation (+db2)	1220.48ms
Memory usage without indexes	750.46 MB = 36.20 MB + etc
Initial indexing	836.29ms
Initial indexing maxcpu=86	4573.27ms
Initial indexing compat	957.05ms
Two indexes updating concurrently	1276.12ms
Key one initial	66.04ms
Key two	1.68ms
Key one again	1.35ms
Reboot and key one again	67.01ms
Latest root posts	1016.88ms
Latest posts	9.68ms
Votes one initial	705.68ms
Votes again	0.67ms
HasRoot	544.10ms
HasRoot again	0.57ms
Author one posts	641.06ms
Author two posts	36.11ms
Dedicated author one posts	661.68ms
Dedicated author one posts again	1.16ms
DeleteFeed	3698.71ms
Maximum memory usage	1019.21 MB = 55.20 MB + etc
Indexes folder size	10.01mb