JSON as a "waist"

[downTheFallLine]

In scope.md, there' a line that says "DIDComm may include formats other than JSON"

Why make this expansion. Seems to me if you focused the effort on building atop JWM you create a "waist" that will simplify discussions and speed adoption. And yes, JWM could map to any number of underlying transport protocols (e.g. SMTP, HTTPS, etc)

[kdenhartog]

As someone who sees the benefits in supporting things like CBOR, I tend to actually see it as beneficial for us to remove this extension point for supporting other representations of DIDComm messages for the reasons pointed out above.

Thoughts on eliminating this @dhh1128 @TelegramSam @OR13 ?

[OR13]

I propose DIDComm implement an abstract data model like did core has.

[tplooker]

I tend to agree with @downTheFallLine because you cannot simply hot swap other data representation formats without defining things such as cipher and digital signature representations (e.g this is what JWS and JWE defines and by association what JWM relies upon) that focusing on JSON is advantageous. I do however like writing the spec as @OR13 suggested which gives it an eye to the future where by DIDComm messages could be ported to another data representation format. The biggest question is about how far we go to keeping this door open in the spec it self, for example if we wanted to entirely future proof DIDComm for a post JSON world we would need to define a way in which for parties of DIDComm messages to negotiate what representation formats they support.

[TelegramSam]

Similar thoughts: focusing on JSON eliminates complications. I'd like a future that allows a more efficient format with additional spec work. As efficiency is rarely a pure show-stopper, I'd rather leave that for future work.

[downTheFallLine]

How does one negotiate representation formats without specifying a representation for the negotiation? @TelegramSam how are you defining efficiency in this context? There's a lot of situations where "wasting" a plentiful resource results in the creation of new innovations.

[dhh1128]

Flexibility in representation is only a virtue if we need it. It has a meaningful carrying cost. Making everything abstract is a mistake I made often as a younger programmer; I was long on theory but had not yet accumulated enough pragmatic battlescars to understand what cost I was incurring. See https://codecraft.co/2012/10/17/flexibility-is-no-virtue/.

We went with an abstract data model for VCs because we clearly needed it. We started out with a concrete data model for DID documents, then shifted to abstract when there was a strong push for it. (I was a proponent of this abstraction, BTW -- but mostly because I detested the concrete model that had been put forth, not because I wanted to facilitate lots of alternatives...) Sorting through the fallout has taken us time and has introduced dozens of github issues and hours of community discussion. My conclusion from this is that if we are convinced we need abstract, we should allow for it early -- but we should not be abstract just because we have a vague notion that it might be a good idea. We should go abstract only if we are confident that it's worth the carrying cost. We can always write a revision of the spec that introduces abstraction later.

So: do we have any compelling arguments for abstract today?

I don't think so. We don't have any existing impls of DIDComm that use an alternative realization, so there is no divergence to accommodate. (That's a very good state of affairs, I think.) Every alternative instantiation of the abstract model that we enable fragments the gravitas of DIDComm ecosystem, making new DIDComm implementations that are conceptually equivalent but not practically interoperable. Why would we do that?

The best alternative repr to JSON, IMO, is CBOR. I like CBOR. It is slick. But is it better enough to justify the carrying cost? Its chief benefit is compactness. But I don't see this as being compelling for DIDComm, since we're generally talking about small messages anyway (though binary attachments would be nice).

[OR13]

(I was a proponent of this abstraction, BTW -- but mostly because I detested the concrete model that had been put forth, not because I wanted to facilitate lots of alternatives...)

I feel similarly about DIDComm choosing to only support JSON, when we know that at scale a pure binary message format would save on storage and enable higher message throughput.

If DIDComm does not support an abstract data model, a CBOR variant of it, that supports COSE will eventually be developed, and it's unlikely they will be interoperable.... folks who really need that performance, will have to support both protocols.

https://www.xml.com/pub/a/2003/02/26/binaryxml.html

The authors of this paper believe strongly that that data and processing model for SOAP has always been and should remain purely XML-based. Literally thousands of man-years have been directed at defining and refining an architecture based on these assumptions. Moreover, the data and processing model for SOAP should deviate as little as possible from the current SOAP/1.2 Candidate Recommendation.

Perhaps its best to think of DIDComm v2 as the SOAP/1.2 CR for JSON + DIDs, and leave things like authenticated CBOR / Protocol Buffers to the next specification.

[downTheFallLine]

This has been a great discussion. I'd invite everyone to think about this, and some of the other issues from the perspective of LEAN 6-SIGMA. In short, beware of over optimizing portions of the design, known as SUBOPTIMIZATION in the nomenclature. Yes, a message in CBOR can be expressed more compactly than in JSON, or XML. This begs the question of the top-level SYSTEMIC goal(s) for DIDCOMM overall. Is the goal to have the most compact, succinct way of expressing messages? Is it to QUICKLY build a community of interoperable implementations? Something else? Then make your tradeoffs accordingly.

[TelegramSam]

@OR13 I like the idea of pushing that to a future version, but I'd feel more comfortable if we had a 'path' there. A way to indicate which version a message uses, so that we have a ready-made identifier to change to signal the next version.

Any ideas on the cleanest way to do that?

[dhh1128]

@TelegramSam : It's easy enough to introduce a header into the JWM profile of DIDComm to express the DIDComm version. The challenge is that if DIDComm v2.0 supports only JSON, and DIDComm v2.3 supports CBOR, then you won't necessarily find a JSON header to help you if you're at v2.3. You need to detect JSON vs. CBOR first (I suggest we do this by checking whether the first byte of the message is {), and then look for a version header next.

[OR13]

this is the exact same issue we have with did core...

content sniffing is generally not a good idea, but we cannot in practice stop it, so its best we define it properly.

in transports that support content-type or similar, I would expect something like this:

application/didcomm+json

application/didcomm+cbor

If there are vendor specific flavors of didcomm, I would expect this:

application/didcomm+vnd.yourcompany.yourproduct-v1.1+json application/didcomm+vnd.yourcompany.yourproduct-v1.1+cbor

These should be registered with IANA.

[llorllale]

Agreed on registering the content types in IANA but DIDComm is explicitly designed not to depend on the transport layer.

Prepending a few bits in front of the didcomm "payload" (where payload is the entire JWE(JWM)) is the best path I can think of that would allow for an easier evolutionary approach.

[kdenhartog]

The problem I've seen when trying to prepend bytes to a JSON object is that:

1. It means we have to do pre/post processing when using a JOSE library straight off the shelf.
2. It means we have to do pre/post processing of JSON to use JSON.parse() in most languages
3. It potentially changes the MIME-type describing the content in the body of a transport layer (e.g. application/json -> application/octet-stream) which can sometimes affect how much a server framework (I'm thinking particularly about HTTP here) will handle for you.

I still prefer to avoid content sniffing and rely on content types to specify the serialization format of the payload. Is anyone using a transport (and would be willing to define the rules to use it) which doesn't have a way to express the type of content in the payload?

[OR13]

thats the thing about transports... they usually have a solution for content representations.... didcomm "is trying to be transport agnostic" is a more accurate description of a WIP spec.

IMO, DIDComm will have lots and lots of transport specific stuff, because thats the way the internet works.

instead of holding an undefined bucket of transports, and saying "didcomm has to support all of these"... it would be smarter to pick a few that will be supported by DIDComm v2, and define support for them exhaustively.

HTTP seems like the first and most obvious one that should be addressed.

[TelegramSam]

DIDComm is transport agnostic in the sense that the main properties (including security) are not derived from the transport layer. The Spec does indeed contain details of proper application to specific transports. This includes the mime type, valid HTTP verbs and response codes, etc.

Detecting outer encoding either involves byte inspection or prepending bytes: both have downsides already articulated here.

HTTP can specify the type. Websockets don't by default but do have two main transport types: text and bytes. This only allows two types to be used if we link them to this, but may be useful: json via text, cbor via bytes.

Perhaps the right compromise here is to lean on transport details when possible, and fall back to defined content detection methods in other cases.

I'm happy to define how this can be done to guide future spec work without actually doing it. This provides a defined path forword, but does not actually step into defining how CBOR would work.

[TelegramSam]

Notes from WG meeting: Daniel is going to test json vs cbor content sniffing.

[dhh1128]

Pseudocode:

• If first three bytes are 0xd9d9f7, assume CBOR. See https://tools.ietf.org/html/rfc7049#section-2.4.5.
• If first 3 bytes are EF BB BF (UTF-8 encoding of the byte-order mark), assume utf8 text and continue.
• If first non-whitespace byte after optional byte order mark is '{', assume JSON.
• else assume unknown format.

Like many sniffing algorithms, this algorithm does not offer iron-clad guarantees of correctness. For example, if someone is trying to encode for gRPC, which uses ProtoBuf encoding, it will currently end up in the "unknown format" bucket. If someone sends CBOR without the special tag, it will also end up in the unknown format bucket. However, the logic is easily described and implemented, is deterministic, and can easily be satisfied by any producer of DIDComm messages.

[llorllale]

I don't like the direction where this is heading. Too ambiguous, clearly not very future-proof. The primary motivation of this exercise was to account for CBOR, and not even that is being guaranteed.

If we're already doing content sniffing, we should just prepend some bytes and we get to set the rules. Otherwise, we drop the whole "transport-agnostic" thing and pick the transports we like where we can signal the content-type.

[dhh1128]

Here's actual python code for the pseudocode above, as promised. Save this as foo.py and run pytest foo.py to run the tests.

def sniffdc(bytes):
    tag = bytes[:3]
    if tag == b'\xd9\xd9\xf7':
        return 'cbor'
    elif tag == b'\xef\xbb\xbf':
        bytes = bytes[3:]
    n = 0
    for c in bytes:
        n += 1
        if c not in b'\r\n\t ':
            if c == 123: # {
                if n < len(bytes):
                    return 'json'
            break
    raise Exception('Unknown encoding.')

def assert_unknown(bytes):
    raised = False
    try:
        sniffdc(bytes)
    except:
        raised = True
    assert raised

def assert_known(name, bytes):
    assert sniffdc(bytes) == name

def test_empty():
    assert_unknown([])

def test_random():
    assert_unknown(b'random stuff')

def test_short():
    assert_unknown(b'\xd9\xd9')
    assert_unknown(b'\xd9')
    assert_unknown(b'{')

def test_json():
    assert_known('json', b'{ "name":"John", "age":30, "car":null }')
    assert_known('json', b' \t { "name":"John", "age":30, "car":null }')
    assert_known('json', b'\r\n\n \t { "name":"John", "age":30, "car":null }')
    assert_known('json', b'\xef\xbb\xbf{ "name":"John", "age":30, "car":null }')

def test_tagged_cbor():
    assert_known('cbor', b'\xd9\xd9\xf7')

[kdenhartog]

Thanks for writing this up @dhh1128

I think this is a good enough path forward to at least allow this as an extension, but the statement "if someone is trying to encode for gRPC, which uses ProtoBuf encoding, it will currently end up in the 'unknown format' bucket" leaves me a bit hesitant that we can only end up supporting a single binary format. This leaves a part of me remains a bit uncomfortable in allowing the serialization format to become an extension point, but not enough to the point where I'd object to it being done.

[dhh1128]

Good comments, @llorllale and @kdenhartog .

As I read the CBOR RFC carefully, my mental model changed a bit. CBOR and ProtoBuf are not direct semantic equivalents of JSON as used in DIDComm, although they are close. The difference is that JSON (at least the way we are using it in DIDComm) is a format for both content and the container of that content, whereas CBOR and ProtoBuf are formats of the content only, EXCLUDING the container of the content. This means that it's rational to be able to detect JSON by looking at the outer bytes of a document (e.g., the curly brace). However, it's not reasonable to be able to detect CBOR and ProtoBuf by looking at the outer bytes of a document, because there's no container -- just the bytes of the content that could be contained. There is no header that declares what follows to be CBOR or ProtoBuf. You just have to know it is. I think the reason for this is that CBOR and ProtoBuf are thought of more often as content embedded inside something else, whereas JSON is thought of more often as a whole document {...}.

I can think of several different ways to erase this subtle semantic difference, but right now the one I'm liking is to give all DIDComm messages an explicit container that is not JSON or CBOR or anything else. It could be ultra simple, consisting of a header line (prepending a mime type + CRLF + CTRL+Z to the real content). The values of the mime type would be application/didcomm+json, application/didcomm+cbor, and anything else we dream up in the future. There would be no ambiguity. The package we've described that's based on JOSE would be everything after the CTRL+Z byte. It would mean we'd have to go through a "decoding" step on every DIDComm message, but that step is trivial (just advance pointer to the offset of the CTRL+Z byte). There wouldn't be any overhead or transformation of data.

[OR13]

SOAP was eventually abandoned because people preferred HTTP and JSON.... being agnostic to the point of a confusing spec, is a risk for DIDComm...

Personally, I would rather not hear "DIDComm is transport agnostic" and instead hear "DIDComm has fully documented support for a growing list of transports".... this problem goes away if we respect the features and conventions of transports.

[dhh1128]

@OR13

Personally, I would rather not hear "DIDComm is transport agnostic" and instead hear "DIDComm has fully documented support for a growing list of transports"

The "transport agnostic" verbiage is a formal part of the approved goals in this working group's charter, not just loose language. It's there for a reason. But that doesn't mean we can't align around the general sentiment you're expressing, Orie. I have no problem at all saying that DIDComm can/should/does respect the conventions of every transport that matters. If HTTP wants to express content type in an HTTP header, then by all means we should respect that convention.

The sticking point is that we can't change the format or content of messages to respect those conventions. Not even a little bit. We can absolutely say that when we transmit a DIDComm message over HTTP, we include a Content-Type header -- but we can't take away the self-describing nature of the DIDComm message and rely exclusively on the transport's content description mechanism.

The reason why this is true is not because we're unwilling to be "HTTP native." Rather, it is because we're unwilling to assume that, in the routing journey between point A and point B, only one transport is involved. I'd venture to guess that >95% of all DIDComm conversations in the next 5 years will use HTTP -- but what matters is not whether HTTP is involved, but whether they ONLY use HTTP. It's perfectly possible to have one hop between A and B -- the major one that is visible to the outside world -- use HTTP, yet have a secondary, hidden hop beyond the HTTP service endpoint that uses a message queue or a raw socket. And if we've made the DIDComm messages cease to be self-describing on the theory that we know we're using HTTP and it has a Content-Type header, we've just created a translation/conversion headache for the raw socket or message queue portion of the route.

To me, that's the issue at the heart of the "transport-agnostic" verbiage -- not that we flout transport conventions, but that messages must be fully self-describing so we can mix and match transports without losing information. I need to be able to email a message that then gets POSTed via HTTP, that then gets saved to a file, and have confidence that there will be no information loss, and that this outcome requires no special translation work.

[OR13]

yep, transport agnostic implies => duplicate all properties in the message body that might normal go in a transport message envelope.

so you have on disk:

• message.json
• message.cbor

each contain inside of them content-type (because versioning):

• application/didcomm+vnd.yourcompany.yourproduct-v1.1+json
• application/didcomm+vnd.yourcompany.yourproduct-v1.1+cbor

then when you need to send them, you open up your transport...

map "content-type" and any other fields from the message to whatever the transport does or does not support... and smoosh your payload into the transport.

on the other side, the recipient gets the message, checks that its well formed (which means making sure the transport and all the duplicated fields inside the message match).

If you want the message to go:

HTTP -> UDP -> PROPRIETARY NFC -> EMAIL -> PIDGEON -> HTTP -> UDP, you need to handle re-encoded all parts of the message for each transport, which means allows intermediaries to "translate" didcomm messages, which means that 100% of what is needed for translation needs to be available to the intermediary.... which is why 100% of what would go in the transport needs to be copied in the message (in a protected plaintext field).... and every-time translation happens you need to ensure its done correctly on the send and receive side.... it is totally doable.... just beware that once you sign json, you can't tamper with it... you can only pull parts of it out and sign it again... the same thing is true for CBOR or any other format.

I think its advisable to construct a demo of how this would work for a single message.

1. Party 1 HTTP JSON Message for Party 5, through intermediary Party 2
2. Party 2 converts to CBOR and sends over UDP to Party 3
3. Party 3 converts to TLV and sends over NFC to Party 4
4. Party 4 converts to TEXT and sends over EMAIL to Party 5

DIDComm provides concrete definitions for message formats:

• JSON
• CBOR
• TLV
• TEXT

DIDComm provides concrete definitions for transport formats:

• HTTP
• UDP
• NFC
• EMAIL

DIDComm also defines formally how to convert safely (preserving authentication and integrity) between all message formats.

Since we are starting with JSON / JWE... the next logical step would be to define CWM for CBOR / CWE.... and then show equivalence before proceeding too far "hoping (without proof)" that transport agnostic behavior will be preserved.

[downTheFallLine]

Wow. A lively discussion. At one level, this seems to be centered on the JSON & CBOR alternatives. In my probably naive view, assuming JSON based DIDComm, the idea was to layer on top of JWM. JWM is seemingly a draft level spec, and it's status expired 18Jul20 (https://tools.ietf.org/id/draft-looker-jwm-01.html). While there's the "foundation of sand" concern given the expired status, JWM itself describes a message structure. It does not deal with transport concerns. So to the extent a DIDComm spec is reliant on JWM, seems the transport layer is kind of moot. Having said that, how all of the layers of the DidComm cake lay down on upon the next is clearly important, and I'd previously proposed a separate implementor's guide (e.g., this is how DIDComm + JWM + HTTPS work). Where did the team land on that point? Turning to the question of CBOR, are we proposing CBOR based payload inside JWM? Some other stack? I fully realize we need to be future-proof and all that. I also personally believe that one should avoid specifying the entire stack in a spec like this. At the same time, it is useful (to me at least) to consider a tangible example when noodling through these complex questions. I also understand the arguments to be made in the case of DIDComm meets IoT (thank you Michael Shea). Therefore, in the case of CBOR encoded DIDComm, what would a REPRESENTATIVE stack look like?

[downTheFallLine]

@OR13 You may be on to something. At the end of the day, CBOR is simply a different serialization format. Treat it as such. Following this line of thinking, you focus the spec development work on the semantics and logic of the messaging protocol itself, and then add a section to the spec for how the messages are serialized. DIDComm supports CBOR and JSON. Done and done. This is similar to the approach taken with RFC 7515.

[OR13]

@downTheFallLine yep, today we have:

JSON -> JWE / JWM -> DIDComm

tomorrow we have:

CBOR -> CWE / CWM -> DIDComm

If we don't see native support for a representation, we are just forcing everyone to take JSON and we are treating the other representations as second class citizens... ironically this is exactly why folks like @dhh1128 and @talltree advocated for an Abstract Data Model for DID Core.... you can see why the added complexity is frustrating for developers.

[downTheFallLine]

Seemed to me that RFC 7515 was able to separate out serialization without going all-in on an abstract data model-- perhaps there's a middle ground.

Where do I go to read up on CWE/CWM?

[TelegramSam]

@OR13 I don't see reencoding as a viable option. The message is created by the sender for the ultimate recipient and remains in that form till received. Mediators employed to transmit a message carry it as a payload to a forward request. If the final message is originally created as CBOR, it's gonna be CBOR all the way to the end.

[dhh1128]

Re. the proposal for a demo:

I consider this a waste of time, because I've already done something similar. I wrote an agent that translated between HTTP, SMTP, and file system. It did the kind of translation you'd expect; thread ids showed up in email headers to create email threads, content types showed up in HTTP and SMTP, etc. What it did NOT do is change the encoding of the message anywhere along the line. I don't see any reason why we would want to do that. I agree with @TelegramSam's latest comment, that whatever the message is at the beginning is what it's going to be at the end.

The code was for an early version of DIDComm v1, not for the latest stuff in Aries and not for DIDComm v2. And I haven't maintained it for a while, so it's probably broken now. But the concepts are still the same, and it demos more or less what you are proposing. You can go look at it if you're curious: https://github.com/evernym/q. See the polyrelay.

Regarding the second-class citizens: The fact is that everything other than JSON actually IS a second-class citizen. Not because we don't like it, but because there are no impls. There is actual stuff--quite a lot of it, actually--in production for DIDComm v1. It's 100% JSON. And this makes JSON the incumbent encoding for v2, also. I'm not hostile to other encodings, but I'm only just barely willing to lift a finger to enable them. I'll go as far as to design for the possibility, but I'm certainly not going to write demos to prove anything about them, or spend time elaborating a test suite for them. That's interoperability for theory's sake, not interop that helps real customers. I view new encodings as fragmenting the space and undermining interop and creating new work for everybody, so they're only worth doing when someone proves they need it and are willing to invest.

[TelegramSam]

As reluctant as I was about header bytes on messages, I think that may be the best option. It requires the least up front work and the greatest flexibility going forward.

[downTheFallLine]

I agree with @dhh1128 that where we focus needs to align to real use cases. and for me personally, that's JSON (Not doing IOT at the moment). I'd rather see getting all the other issues raised, and separating out serialization as a separate line of thinking.

[OR13]

The fact is that everything other than JSON actually IS a second-class citizen. Not because we don't like it, but because there are no impls. There is actual stuff--quite a lot of it, actually--in production for DIDComm v1. It's 100% JSON.

@talltree I hope you can appreciated the irony of reading this, based on the Abstract Data Model conversations in DID Core.

FWIW I actually agree, the standards should follow what implementers actually support, and I think its correct to just say: "DIDComm is a JSON-only message oriented set of protocols, that is transport agnostic, assuming your transport can handle sending some form of JSON or encoded JSON".

Its for the exact same reason I argued that "JSON-only" DID Documents should not be supported, since there were no implementations of them until I made https://did.key.transmute.industries....

I suggest not doing any fancy "padding" / encoding... and instead just define a single content-type:

appliation/didcomm+json and then allow each transport to define how it handles that content type.

No need to support COSE/ CBOR / Protocol Buffers, and no need to play games prefixing buffers so that content can be sniffed.

[dhh1128]

(I deleted my previous comment at this point in the stream. I decided that, although it wasn't super cranky, it wasn't constructive. :-)

[kdenhartog]

It could be ultra simple, consisting of a header line (prepending a mime type + CRLF + CTRL+Z to the real content). The values of the mime type would be application/didcomm+json, application/didcomm+cbor, and anything else we dream up in the future. There would be no ambiguity.

This isn't "beginner developer friendly", but it's so close to it that I think it will work. +1 from me

One question worth considering while we're on this topic. Let's say we're on a transport that has a 1000 byte limit and the didcomm message is 1456 bytes. Would we want to prepend this onto both messages, maybe add an additional "message one of 2" byte on there, or potentially just say this isn't a good fit for DIDComm?

Also, on the topic of not changing serializations, my thinking is that something like this is could be needed for massive mediators. However, while I can come up with a good reason that I think this would be needed, I want to avoid going down this rabbit hole further and say we'll deal with that in a later version so we can focus on the more pressing problems we need to in order to get DIDCommV2 stabilized.

My preference would be to move Daniel's idea into the specification and defer the side discussions that have popped up into discussions for later versions of this work and then declare this resolved. This is mainly because I'd like for us to not get hung up on this topic much more since it's just a "future proof feature".

[OR13]

@kdenhartog are you basically just saying "lets get JSON correct first" ? I am +1 to that.

[TelegramSam]

Here is a super simple strategy I think we should use: Future formats are responsible for identifying themselves via inspection of beginning bytes. This can be a string as discussed, or (for something like CBOR) a very identifiable tag. If no bytes are found, this content will be JSON. This allows us to make no changes now, AND have a way to add them when useful in the future.

[OR13]

@TelegramSam why not just say: "Future content types will be supported", and not define any part of that process.... I am generally not a fan of "inspecting bytes as a way of sniffing content".... its a security vulnerability on a plate.

• https://www.checkbot.io/guide/security/content-sniffing/
• https://en.wikipedia.org/wiki/Content_sniffing
• https://hackerone.com/reports/363845

.... and since we know we are not going to actually support CBOR / Protocol Buffers any time soon... better to not hint at an insecure mechanism for supporting them, which we are not actually going to be using any time soon.

All this to say: "YES" JSON for now, and maybe JSON forever!

[downTheFallLine]

i totally agree @TelegramSam content sniffing is not a good way to proceed in my view.

[TelegramSam]

An identifiable string hardly qualifies as content sniffing. The CBOR tag is designed precisely of having an easy way of identifying CBOR, and any prefixed string is the same as an inline content-type header. Let's discuss the right language to use in the spec on the call today.

[OR13]

unfortunately, i can't make the call today, but my advice is not to define "future proofing support for alternative representations" without actually defining "alternative representations"... not a hill I would die on though.

[TelegramSam]

https://github.com/decentralized-identity/didcomm-messaging/pull/122

[dhh1128]

Per this comment from @OR13, I've raised a PR clarifying MIME types. If we can agree to it, I will register them with IANA as well.

[TelegramSam]

Close in WG Meeting 2020-11-02