Change the serialization to CBOR

decentralgabe commented 1 month ago

Inspired by https://github.com/TBD54566975/did-dht/pull/264

CBOR is much simpler and more efficient. There is wide support for it across languages.

We can also look into:

The Plain CBOR Representation v1.0 from the DID WG
CBOR-LD 1.0 from the JSON-LD org
dCBOR: A Deterministic CBOR Application Profile representation in the IETF

I am tagging @msporny and @christophera both of whom I've spoken with about adopting CBOR. It would be great to hear your perspectives and considerations (Manu, I know you raised some great points about versioning and not conflicting with already registered terms).

ChristopherA commented 1 month ago

The current "Plain CBOR Representation" is a terrible example of CBOR. It basically base-64 encoded text from a JSON DID Document, which is embeds in AS ENCODED TEXT, not a binary encoding, into a number of text CBOR tags. This results in many of the benefits of CBOR being lost. It may also not be deterministic (i.e. different CBOR encoders will have different output given the same input).

We have a general article on "Why CBOR?" (and a video). The article also has links at the bottom with a comparison to Protocol Buffers, a comparison to Flatbuffers, and a comparison to other binary formats.

At minimum, any implementation of DID Documents in CBOR needs to take full advantage of what CBOR offers.

I don't think it would be that difficult to craft a fully-tagged dCBOR (deterministic CBOR, a constrained version of CBOR, see dCBOR overview) for use by non-JSON-LD DID Documents. There are dCBOR libraries for many languages. We basically take the abstract data model and carefully implement it with best CBOR practices. Blockchain Commons also have already registered a number of tags with IANA for binary versions of public keys and signatures as part of our other work with the IETF, and understand the issues of adding the others required for DID document interoperability.

In some ways this is the simplest solution to offering a "plain" CBOR-based DID Document.

However, there are some real advantages to considering using Gordian Envelope (top page, executive summary, features, internet-draft, video "understanding Gordian Envelope Pt 1.) for DID Documents, but it is more complex spec, and unlike dCBOR, is not yet standards track at IETF.

Gordian Envelope already is on top of dCBOR, has multiple libraries (Rust and Swift), tooling (cli apps), already offers a triple-store, but most importantly, offers elision and compression. This means a DID document could be as small as a signed hash commitment to a tree of elided data. You can see an example of this in the "herd privacy" section of our Educational Use Case for Gordian Envelope: https://github.com/BlockchainCommons/Gordian/blob/master/Envelope/Use-Cases/Educational.md#part-three-herd-privacy-credentials

{
    ELIDED [
        ELIDED: ELIDED [
            ELIDED (5)
        ]
        ELIDED (3)
    ]
} [
    verifiedBy: Signature
]

Given this fully elided envelope in a DHT (basically the size of 6 SHA-256 hashes, 1 signature, plus some overhead for tags, ~<256 bytes), any elided sub-graph of the DID Document be proven by an out-of-band inclusion proof (video), which is also a Gordian Envelope. We already have developers using Gordian Envelope for airgap (QR code) and NFC (on java cards) solutions, which also are very constrained.

You don't necessarily need to fully elide a DID Document, but I wanted to demonstrate that the fully-elided form can be very concise and useful.

-- Christopher Allen

cc/ @wolfmcnally @shannona

msporny commented 1 month ago

I expect that there are a few paths forward for did:dht:

Convince the DID WG to create a bespoke/artisnal encoding of CBOR for DID Documents. This wouldn't take a lot of work, wouldn't work well with extensions, but could do quite a bit of compression on basic DID Documents (including did:dht documents).
Re-use CBOR-LD for compression, which gets you 90% of the compression of the option above and it works for extensions.
Figure out a way to point to external resources from did:dht and store the DID Documents at the external location (and obviate the need for CBOR).

I think I like the third option the best, but it does make a look up have to go to two locations. That said, I think you avoid the need to have to compress to CBOR at all if you go that route. Have we considered using a VC on the mainline DHT entry to point to an external resource (which would be the DID Document)?

Something like this:

did:dht: -> mainline-dht-entry(VC pointing to DID Document on one or more endpoints) -> DID Document sitting on regular 'ol website?

I know we lose storing the DID Document on the DHT, but maybe its worth it to not have an upper bound on a DID Document (and stuff like did auditing support)?

Just some very loose, disconnected thoughts. I see all three options as viable, and options that could be done in parallel if we put a header on the did:dht value or in the entry in the DHT (to say how to do the rest of the look up). The options could be: 1) the data is here and is compressed in artisnal CBOR, 2) the data is here and compressed in CBOR-LD, and 3) the data is elsewhere, here's one or more digitally signed pointers to the data (as a VC)

TBD54566975 / did-dht

Change the serialization to CBOR #293