Supported public key formats?

[msporny]

@selfissued wrote:

We should be removing public key formats from the spec - not adding them. We should get down to one key format to increase interoperability of implementations - preferably JWK.

We should decide (as soon as possible) whether we will be supporting exactly one or more than one key expression format in the DID specification.

Options include: JWK, COSE Key, LD Keys, PEM, or expression of key-like data structures (for things like ethereumAddress).

We will need at least @awoie, @selfissued, @gjgd, @dlongley, @ChristopherA, and @csuwildcat to participate in this discussion.

Related to PRs #56, #63 and issues #55, #24, #20, #15, and #7.

[msporny]

We should start the issue off w/ examples of how various types of keys can be expressed using the options above.

[selfissued]

For us to have a well-informed discussion on this topic, it would be useful to have an inventory of the different kinds of public keys that we will want to be able represent. I'm aware of (1) RSA public keys, which need a mantissa and an exponent, (2) elliptic curve keys that need an X and Y pair, and (3) elliptic curve keys that only require an X value (all of which have standard JWK representations).

Are there different kinds of public keys we care about that aren't of any of the kinds 1-3 above? If so, what values do their representations contain? (For extra credit, please provide specification references so that we can all get up to speed them :-) .)

[ChristopherA]

(2) elliptic curve keys that need an X and Y pair, and (3) elliptic curve keys that only require an X value (all of which have standard JWK representations)

No, X and Y is not correct. What is needed is the finite field form (in the case of bitcoin's secp256k1 it is Galois y² = x³ + 7), the finite field's modulus (usually called p, which in secp25k1 is p = 2^256 – 2^32 – 977 which makes this a Kobliz Galois field), and a generator base point (usually called g, which in secp256k1 is TWO numbers, 0279BE667EF9DCBBAC55A06295CE870B07029BFCDB2DCE28D959F2815B16F81798, or in “compressed form” 040x79BE667EF9DCBBAC55A06295CE870B07029BFCDB2DCE28D959F2815B16F81798483ADA7726A3C4655DA4FBFC0E1108A8FD17B448A68554199C47D08FFB10D4B8). 

25519 is quite a bit more complicated than that — it is a Montgomery curve in one form EdDSA and a twisted Edwards in other form Ed25519. Not all public keys are even allowed — there are 11 parameters that define 25519! https://medium.com/@LongHashDot/the-parameters-of-ed25519-fd7da1c0e8d4 And this is an IETF standard. Unfortunately, 25519 has problems with multisig, so there is a variation called Ristretto that isn't compatible with the standard though technically most of the parameters are the same. https://ristretto.group/why_ristretto.html

It gets much worse with signatures, where there can be multiple signature approaches for the same public/private key pair.

I don't consider myself a cryptographer any longer — talk to an active practitioner, but saying they can be defined easily with an x/y pair worries me in its simplification.

— Christopher Allen

[selfissued]

You're of course right, Chris, that the underlying representations and parameter sets can be wickedly complicated. And that the two values used for secp256k1 do not represent a curve point, etc. (As another example of underlying complexity, efficient representations of RSA private keys can use 6 or more values, per https://tools.ietf.org/html/rfc7518#section-6.3.2 ).

And you're of course right that multiple signature algorithms can be used with a given key representation - which is why we also want to carry the algorithm identifier with the key.

Anyway, as a practical matter, I'm trying to have us inventory the kinds of public key representations that we're likely to need. As I'd written to Manu privately, if new kinds of representations are needed, I personally commit to quickly creating specifications to standardize their JWK representations and shepherding them through the IETF process - just as I created https://tools.ietf.org/html/draft-ietf-cose-webauthn-algorithms-01 to standardize identifiers needed for W3C WebAuthn (and for DIDs!). In my experience, if a W3C working group needs something from the IETF, the IETF is more than happy to oblige.

[ChristopherA]

A segue,

I have an odd emerging one for my wishlist (@selfissued, don't take this as a request for your IETF list yet, as it is only emerging), which is colloquially known as secQ. It basically is special mirror curve to secp256k1 that allows for certain kinds of advanced proofs.

See https://github.com/BlockchainCommons/secp256k1/issues/1#issuecomment-410482607 for an overview, and for the kinds of advance proofs it can do: https://electriccoin.co/wp-content/uploads/2019/09/Halo.pdf, the authors of which say that secQ may be a better approach for the technique in the curve for their paper at https://twitter.com/ebfull/status/1171531807105282049?s=20

It is emerging things like Halo, zk-snarks, etc. that is part of why I have been urging this community to use the term "proofs" rather than "signatures" as a classic digital signature is only one kind of proof.

-- Christopher Allen

[peacekeeper]

Yes it would be nice to resolve this long-standing topic. Some additional things to consider:

1. Another option that has been proposed is to use publicKeyMultibase or publicKeyMulticodec for all keys.
2. Independently of what the decision is going forward, it would still be nice to have a JSON-LD context hosted somewhere that simply reflects the CGFR of the spec. This could be done immediately. I believe this was the original intention of this PR.
3. After making a decision, we may then want to reconsider the role of the Linked Data Cryptographic Suite Registry as well as several open PRs there (https://github.com/w3c-ccg/ld-cryptosuite-registry/pull/14, https://github.com/w3c-dvcg/lds-ecdsa-secp256k1-2019/pull/2).

[dlongley]

For us to have a well-informed discussion on this topic, it would be useful to have an inventory of the different kinds of public keys that we will want to be able represent.

I do think that's useful. But I think we'll find (and this thread thus far has highlighted) that the real answer here is "We don't know".

Whatever we pick today may change tomorrow. We don't want to throw up any undesirable roadblocks that require everyone to go through a heavy-weight standardization process in order to put a key into their DID Document. While it is admirable to see an individual commit to helping ensure any newly desired key format will become a standard via IETF, we should not rest our entire strategy for key extensibility on a single individual or group; rather we want to support innovation and organic adoption and consensus at the edges.

This is precisely one of the reasons we've chosen a generalized approach for the data model that supports decentralized extensibility. We have a graph-based data model that can be understood in the abstract without a specific syntax -- and it can be independently extended using standards such as JSON-LD, without asking for permission. We must, therefore, ensure that whatever key representation mechanism we use does not break from that model.

Having "one way" to do things is important, but not as important as extensibility. Taken together, it is most important to have one way of doing extensibility -- which we already have today in the spec through Linked Data. We want to give people space to innovate and continuously evolve -- and let further standardization happen when there is popular uptake.

So let's be clear that our goal (here with keys and elsewhere) is not to make a perfect, beautiful spec that has everything in it anyone might ever need, and if not, they can just ask a kind person who has promised to help. The goal is to get interop where we have consensus today and enable people to continue to innovate independently, within an extensibility framework that can lead to future standardization if desirable. This necessarily means adopting mechanisms that enable decentralized innovation. This is also in the spirit behind DIDs: recognizing independent existence.

So, if we need "one way" to represent keys, let's fit it into our existing model. This is why there is currently support and examples in the spec for LD keys: they fit into this model. While there is an LD key registry, a key needn't be on that registry to be used or expressed. You can independently define your own key terms and @context to map them to globally unambiguous semantics without anyone's permission.

We should also keep in mind that this extensibility framework helps when interop on specific key formats largely happens at the application level in a way we can't necessarily easily influence (where CBOR/COSE, etc. formats may be used and we want to include these keys in a DID Document). This fact also hints that it may be the case that it's not worth the trouble to convert specific key types to "one true format". It may be less trouble to just represent them in their common forms, using this extensibility framework, as best we can in a way that naturally fits with the particular application/platform that they are likely bound to (or should be bound to) anyway. This is the approach that was taken with publicKeyPem for example, rather than converting keys to JWK; as many applications accepted PEM, not JWK.

[OR13]

My preference is to have official support for publicKeyPem and publicKeyJwk, at least, because these will enable wide interoperability with legacy crypto systems. I think If people want to extend the DID Document context to support Paillier or other crypto, or non publicKey like entries such as ethereumAddress. I think these extensions should be made explicit, through the addition of other contexts.

Here is an example DID that contains a bunch of different key types:

https://uniresolver.io/#did:btcr:xxcl-lzpq-q83a-0d5

Including publicKeyPgp which I am still trying to formalize to add OpenPGP support to the ecosystem. I think its reasonable that the DID Spec might not want to support every possible key encoding, especially experimental ones... DID Method implementers should be responsible for defining custom properties that are not in the official did context. like so:

 "@context": [
    "https://schema.org/",
    "https://w3id.org/security/v1",
    "https://w3id.org/did/v1",
    "https://example.com/did-openpgp/v1"
    "https://example.com/did-ethereum/v1"
  ],

On a related note, I have used tools like https://github.com/EternalDeiwos/keyto#usage

To automatically convert keys of formats I can't use (with JOSE for example) to formats I can use.

One of my least favorite parts of converting keys before using them is that the publicKey id property is often not in anyway connected to a reasonable thumbprint, such as:

https://tools.ietf.org/html/rfc7638

or GPG thumbprint or openssl fingerprint...

When handling a credential or ciphertext, its very helpful if the key identifier is well defined according to an RFC... key formats that don't have a standard thumbprint should be discouraged from inclusion in the standard context IMO.

I'd love to see some SHOULD language around key identifiers that encourages something like this:

{
      "id": "did:btcr:xxcl-lzpq-q83a-0d5#key-JUvpllMEYUZ2joO59UNui_XYDqxVqiFLLAJ8klWuPBw",
      "type": "EcdsaSecp256k1VerificationKey2019",
      "publicKeyJwk": {
        "crv": "secp256k1",
        "kid": "JUvpllMEYUZ2joO59UNui_XYDqxVqiFLLAJ8klWuPBw",
        "kty": "EC",
        "x": "dWCvM4fTdeM0KmloF57zxtBPXTOythHPMm1HCLrdd3A",
        "y": "36uMVGM7hnw-N6GnjFcihWE3SkrhMLzzLCdPMXPEXlA"
      }
    },

Note the reuse of the standard kid per rfc7638.

I would hope any key where a thumbprint/fingerprint is common would leverage that identifier to make lookups in the DID Document easier.

[gjgd]

Agree with @dlongley and @OR13

Even if we successfully put together an exhaustive list all key formats being used in all DID methods today (and that would be very hard), this list would still be rigid to the use of future key formats, and supporting those new formats would involve creating a new version of the spec every time.

Using something like publicKeyMultibase would certainly be an improvement but would not solve the problem for all key formats since they can vary along other axis besides the base of the encoding (eg did:ethr's ethereumAddress or @OR13 's example above ^)

We need a more flexible way of describing the key format. I second @dlongley's proposition to leverage the JSON-LD structure of the DID Document, and embed the format of the key in the key object itself using a @context

[selfissued]

Thanks @dlongley etc. for the useful thoughts on the kinds of extensibility that we want to have. I 100% support the goal of enabling new key representations and algorithms to be added by anyone when needed.

JWKs do enable extension by anyone through use of the IANA registries "JSON Web Key Types", "JSON Web Key Elliptic Curve", and "JSON Web Key Parameters" - all at https://www.iana.org/assignments/jose/jose.xhtml . Per the registration instructions at https://tools.ietf.org/html/rfc7517#section-8, any specification can register values. It doesn't have to be an IETF specification, or indeed, even a specification in a standards body. (For example, the W3C Web Crypto API took advantage of this flexibility by registering algorithm and key values at https://www.w3.org/TR/WebCryptoAPI/#iana-section ).

So please don't take my volunteering to be a proactive spec writer as we need new representations and algorithms as a sign that we would be dependent upon one individual or one organization. Anyone can do this at any time - by design.

As a practical engineering matter, our implementations will be simpler and more interoperable if we choose a standard key representation for key types that we already know about. There are dozens of JWK implementations in every conceivable language. For a very partial list, see https://openid.net/developers/jwt/ . And as I discussed with Manu offline, JWK (and JWS, JWE, and JWT) were intentionally designed to only require JSON representations, bringing us into the modern programming world. That way there would be no requirement for implementations to also support other formats such as ASN.1, CBOR, etc.

My goal here is to keep simple things simple. Developers will thank us for years if we do.

[TallTed]

Interop is best improved by allowing for and encouraging lossless translation between data representation formats, not by forcing a single format (which may later be discovered to be problematic).

[dlongley]

@selfissued,

As stated above, I'm onboard with the argument that having a single way to do things where it will help with interop. But when we're talking about key formats, the main point of interop is at the application level. What formats are applications expecting? Will choosing "one true format" actually help with interop? Note that, over in our corner here, we do not have control over the rest of the crypto ecosystem. We shouldn't pretend that if we adopt JWK as the "one true format" for keys in DID Documents that it will mean that all of the various applications and platforms out that don't support JWK will suddenly add support. In fact, as we've mentioned above, some new applications/platforms are adopting COSE/CBOR style key formats despite JWK already being available as an option.

Sadly, what this means is that if DID Documents were to adopt JWKs as "the one true format" then we'd all be having to write key format conversion code everywhere! It would be much better to define some simple (and lossless) encoding rules that takes key formats in their popular forms and minimally encodes them for storage in Linked Data JSON. This is what the community has done thus far with things like publicKeyPem, etc. While this does result in more than one way of representing a particular key (unfortunate), it does mean that applications don't need a JWK to PEM/DER conversion tool, they can just pass the data right to their application (which only understands PEM/DER). This approach has thus far proven to be a lower barrier to get interop.

I agree that it is unfortunate that there are so many ways to represent the same keys. But we must admit that to a large extent, this is beyond our control. Trying to pick a winner here will only result in a loss. We should do something much simpler -- which I believe the community has already been doing. That doesn't mean that we shouldn't try to unify encoding formats where we can by using, for example, multibase. And we certainly should support JWK keys as some applications do accept these. I'd rather not have to convert there either. But adding an extra layer to convert all of the various formats to/from JWK is undesirable -- and I think that's exactly what we'd end up with because we don't control and can't easily influence the entire crypto application/platform space.

[dlongley]

Note that a DID Document may express many different keys for many different intended purposes; only some subset may be used to establish control over modifying the DID Document itself (and would therefore be consumed in some fashion by the DID network/ledger according to the DID method). Others are often application/platform specific. This requirement may not have been understood before by everyone participating in the issue and could help explain why "one true format" isn't the best fit here.

[ChristopherA]

Re: Application Level Support

There may be a number of different kinds of keys that should reside in a DID Document that are not used to validate the DID Document itself. In particular, keys in a number blockchain use cases may be shares of keys or partial keys to be used in multisig scenarios, or are proofs of key possession, such as a hash of a key (for instance a bitcoin address is a base58 hash of the public key). I worry that specifying JWK which does not support these kinds of keys will mean that those applications will not be able to use DID Document to share key information.

-- Christopher Allen

[selfissued]

We had a useful clarifying discussion on this topic on today's call. I asked why people were talking about keys for applications when the keys being discussed are used by central resolvers to validate control of the DID. @dlongley clarified that some of the keys in a DID are application keys and not used for DID authentication, which is a useful distinction to have in mind.

As a result of today's discussion, I think that this issue is really two issues:

1. What format(s) are supported for the public keys used to prove control of a DID?
2. What format(s) are supported for the public keys that are application data?

Having a standard representation for (1) public keys used to prove control of a DID would give us efficiencies both in implementation size and spec size. Standards are about making choices to improve interop and this is place we should choose.

Applications (2) can represent data however they want. That's not the business of the DID standard. What is critical for the standard to do, however it to make it syntactically clear which keys are of type (1) and which data structures are application data structures, including applicaiton data structures that may or may not represent keys.

[dlongley]

@selfissued,

Applications (2) can represent data however they want. That's not the business of the DID standard.

+1

What is critical for the standard to do, however it to make it syntactically clear which keys are of type (1) and which data structures are application data structures, including applicaiton data structures that may or may not represent keys.

I actually think this belongs in the domain of specific DID methods. A particular DID method may certainly say that JWK must be used for that purpose (and should if it makes sense). But I think it would be a bit odd to say all DID methods must use JWK to express the keys that are used internally to determine control over modification of a DID Document. These rules are entirely wrapped up in a particular method and the method should have full leeway to express keys in whatever way is most appropriate for the network/ledger/mechanism used to implement the method.

[iherman]

This issue was discussed in a meeting.

• No actions or resolutions
  View the transcript

  Ivan Herman: See Issue #67
  Michael Jones: As one of the assignees, I have taken the view that key types that the json web key is already a standard.
  … There are other fundamental issues being raised. Keys in the DIDs are being used to check signatures.
  … These are used by applications, so we should cater to the appliations. It seems that having only one way to do it would increase interop.
  Manu Sporny: I think we should put in a couple of examples.
  … We have requirements that are hinted at, but are not being written down.
  … Other views need to be clearly articulated in the issue discussions.
  … Please add your views to the conversation.
  Michael Lodder: agreed
  Manu Sporny: We need a full hour to discuss this.
  … In DID resolution, an idea that an option could be passed to the resolver to request keys in a particular format.
  Markus Sabadello: There needs to be more discussion about this. There is the linked data signature. There is multibase.
  Michael Lodder: I’m concerned we are trying to force an encoding when we should allow the encoding in value
  Michael Lodder: +1 markus_sabadello
  Manu Sporny: I agree with mike-lodder
  Manu Sporny: (based on the requirements we’ve heard)
  Michael Jones: Dave wrote in the issue, that the keys are used by the DID resolver not an application level thing.
  Dave Longley: The keys may not be verified by the resolver. Some keys can be bound to an application.
  … Some keys can be used by the resolver, others by applications.
  Dave Longley: selfissued: in short, a DID Document may express many different keys for many different intended purposes, only some subset may be used to establish control over modifying the DID Document itself
  Dave Longley: (and would therefore be consumed in some fashion by the DID network/ledger according to the DID method)
  Markus Sabadello: +1 to dlongley, keys in a DID Document may or may not be used by a method and the resolver.
  

[selfissued]

To the extent that DID methods check signatures with public keys to validate control of the DID, I sure hope that we architect the data structures so that the signature checking can occur in a way that is independent of the DID method. Otherwise, interoperable implementations would have to implement the ever-increasing union of everything that developers might invent over time - a moving target that would never be hit in practice.

Making choices has clear benefits. (That's why we make standards!)

[OR13]

publicKeyPem and publicKeyJwk would seem to solve for application level interoperability, I'm not sure there is tremendous benefit from mandating one over the other. Because of the success of OpenID and OpenSSL I would advocate for supporting both.

Regarding did document signing or self certifying cryptographic dids, that is where I feel we should rely on method specific extensions, blockchain specific signatures or publicKey like properties such as ethereumAddress, are unlikely to assist with interoperability, and so it feels appropriate for their support to be up to method implementers.

If I want to check an ethereum signature for did:ethr and I am normally using did:btrc, I will need to account for ethereum address transformation of public keys... should every DID Method have information related to this process embedded in the shared context?

[msporny]

This comment attempts to gather feedback across various communities and:

1. List the use cases for expressing public keys in DID Documents.
2. List the derived requirements.
3. List the concrete implementation options under consideration by the group.
4. List the requirements achieved by each implementation option.

Use Cases

The following is a list of use cases for expressing public keys (and other verification methods) in DID Documents.

1. Express public keys such that it may be read by any application connected to the Internet.
2. Express public keys in a way that enables metadata to be associated with the keys (controller, description, labels, etc.) in a semantically deterministic and decentralized way.
3. Express public keys in a way that minimizes the encoding burden on the expected developer audience.
4. Express public keys in a way that minimizes the decoding burden on the expected developer audience.
5. Express public keys in a way that minimizes the encoding/decoding options.
6. Express public keys in a way that is aligned with other standards such as WebAuthn.
7. Express verification methods in a way that allows for non-public key mechanisms to be utilized (such as Bitcoin P2SH, ethereum contract addresses, proof of stake, proof of elapsed time, etc.)
8. Express verification methods (of which public keys are a subset) in a way that enables M-of-N usage scenarios.

Note that the last two use cases, while not specifically public key use cases, are related such that the mechanism for expressing verification methods should ideally be able to support public keys and other verification methods.

Requirements

The following requirements are derived from the use cases above.

1. Public keys MUST be readable by clients connected to the Internet.
2. Metadata MUST be able to be associated with public keys (and verification methods in general). It MUST be easy to extend the types of metadata associated with a public key in a decentralized way.
3. An expected developer SHOULD be able to encode a public key using minimal tooling.
4. An expected developer SHOULD be able to decode a public key intended for their use/environment with minimal tooling.
5. The encoding/decoding formats SHOULD be minimized to the greatest extent possible while supporting as many use cases as possible.
6. The encoding/decoding mechanism SHOULD support the native datatypes used in WebAuthn.
7. The verification method expression mechanism MUST support non-public key approaches to cryptographic security.
8. The verification method mechanism MUST support M-of-N usage scenarios such as multisig.
9. The solution should use existing well defined and ratified standards.
10. The solution should be implementable using widely deployed tooling.
11. An implementation that can check signatures/proofs for a particular key type will be guaranteed to keep working for all future instances of that key type, because alternate representations requiring additional code to translate key formats will not be introduced.

Implementation Options

I'm going to list all of the options that we are seriously considering so that people can see exactly what some of these decisions might look like. Some of the people in this thread are new to the topic, so seeing concrete examples might help more people contribute to the discussion. I'm not listing every option because I started to, got to 15 different representations, and felt that would further muddy the waters. If folks feel that there are other serious contenders, please speak up and I'll add them to the list below.

Option A: Pure JWK

"authentication": [{
  // RSA public key expressed in component form
  "kty":"RSA",
  "n": "0vx7agoebGcQSu...DKgw",
  "e":"AQAB",
  "alg":"RS256",
  "kid":"2011-04-29"
}, {
  // Elliptic Curve P-256 public key expressed in component form
  "kty" : "EC",
  "crv" : "P-256",
  "x"   : "SVqB4JcUD6lsfvqMr-OKUNUphdNn64Eay60978ZlL74",
  "y"   : "lf0u0pMj4lGAzZix5u4Cm5CMQIgMNpkwy163wtKYVKI",
  "d"   : "0g5vAEKzugrXaRbgKG0Tj2qJ5lMP4Bezds1_sTybkfk",
  "kid":"2015-02-13"
}, {
  // Ed25519 public key expressed in component form
  "kty" : "OKP",
  "crv" : "Ed25519",
  "x"   : "11qYAYKxCrfVS_7TyWQHOg7hcvPapiMlrwIaaPcHURo",
  "d"   : "nWGxne_9WmC6hEr0kuwsxERJxWl7MmkZcDusAxyuf2A",
  "use" : "sig",
  "kid" : "FdFYFzERwC2uCBB46pZQi4GG85LujR8obt-KWRBICVQ"
}]

Option B: Linked Data + various "native" key formats

"authentication": [{
  // RSA public key expressed in PEM format
  "id": "#keys-1",
  "type": "RsaVerificationKey2018",
  "controller": "did:example:123456789abcdefghi",
  "publicKeyPem": "-----BEGIN PUBLIC KEY...END PUBLIC KEY-----\r\n"
}, {
  // RSA public key expressed in JWK format
  "id": "#keys-2",
  "type": "RsaVerificationKey2018",
  "controller": "did:example:123456789abcdefghi",
  "publicKeyJwk": {
    "kty":"RSA",
    "n": "0vx7agoebGcQSuu...JzKnqDKgw",
    "e":"AQAB",
    "alg":"RS256"
  }
}, {
  // Elliptic Curve Secp256k1 public key expressed as COSE Key (base64url encoded)
  "id": "#keys-3",
  "type": "EcdsaSecp256k1VerificationKey2019",
  "controller": "did:example:123456789abcdefghi",
  "publicKeyCose": "pSABIVggusWxHK2PmfnHKwXPS54m0kTcGJ90UiglWiGahtagnv8iWCAgE4v4LcG21WK-D6VKt4BKOmS21yzP7Wtvtu0ou_wRfgECAkIxMQ"
}, {
  // Ed25519 public key expressed in base58btc format
  "id": "#keys-4",
  "type": "Ed25519VerificationKey2018",
  "controller": "did:example:pqrstuvwxyz0987654321",
  "publicKeyBase58": "H3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV"
  },  {
  // Elliptic Curve Secp256k1 public key expressed in hex format
   "id": "#keys-5",
   "type": "EcdsaSecp256k1VerificationKey2019",
   "controller": "did:example:123456789abcdefghi",
   "publicKeyHex": "02b97c30de767f084ce3080168ee293053ba33b235d7116a3263d29f1450936b71"
}, {
  // Elliptic Curve Secp256k1 public key expressed as an Ethereum address
   "id": "#keys-6",
   "type": "EcdsaSecp256k1VerificationKey2019",
   "controller": "did:example:pqrstuvwxyz0987654321",
   "ethereumAddress": "7F01A23DB00E522f8871F3E56d669467EED10368"
  }, {
  // Elliptic Curve Secp256k1 public key expressed as an Bitcoin address
  "id": "#keys-7",
  "type": "EcdsaSecp256k1VerificationKey2019",
  "controller": "did:example:pqrstuvwxyz0987654321",
  "bitcoinAddress": "1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2"
}]

Option C: Linked Data + various "native" key formats + Multikey (Multibase + Multicodec)

"authentication": [{
  // RSA public key expressed in PEM format
  "id": "#keys-1",
  "type": "RsaVerificationKey2018",
  "controller": "did:example:123456789abcdefghi",
  "publicKeyPem": "-----BEGIN PUBLIC KEY...END PUBLIC KEY-----\r\n"
}, {
  // RSA public key expressed in JWK format
  "id": "#keys-2",
  "type": "RsaVerificationKey2018",
  "controller": "did:example:123456789abcdefghi",
  "publicKeyJwk": {
    "kty":"RSA",
    "n": "0vx7agoebGcQSuu...JzKnqDKgw",
    "e":"AQAB",
    "alg":"RS256"
  }
}, {
  // Elliptic Curve Secp256k1 COSE Key expressed as a Multikey
  "id": "#keys-3",
  "type": "EcdsaSecp256k1VerificationKey2019",
  "controller": "did:example:123456789abcdefghi",
  // base64url encoded COSE Key (0xcc as byte identifier for COSE Key)
  "publicMultikey": "uzKUgASFYILrFsRytj5n5xysFz0ueJtJE3BifdFIoJVohmobWoJ7_IlggIBOL-C3BttVivg-lSreASjpkttcsz-1rb7btKLv8EX4BAgJCMTE"
}, {
  // Ed25519 public key expressed as a Multikey
  "id": "#keys-4",
  "type": "Ed25519VerificationKey2018",
  "controller": "did:example:pqrstuvwxyz0987654321",
  // base58btc encoded Ed25519 raw bytes (0xed as byte identifier for ed25519 public key)
  "publicMultikey": "zH3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV"
  },  {
  // Elliptic Curve Secp256k1 public key expressed as a Multikey
  "id": "#keys-5",
  "type": "EcdsaSecp256k1VerificationKey2019",
  "controller": "did:example:123456789abcdefghi",
  // base16 encoded Secp256k1 (0xec as byte identifier for Secp256k1 public key)
  "publicMultikey": "fec2b97c30de767f084ce3080168ee293053ba33b235d7116a3263d29f1450936b71"
}, {
  // Elliptic Curve Secp256k1 public key expressed as an Ethereum address
  "id": "#keys-6",
  "type": "EcdsaSecp256k1VerificationKey2019",
  "controller": "did:example:pqrstuvwxyz0987654321",
  "ethereumAddress": "7F01A23DB00E522f8871F3E56d669467EED10368"
  }, {
  // Elliptic Curve Secp256k1 public key expressed as a Bitcoin address
  "id": "#keys-7",
  "type": "EcdsaSecp256k1VerificationKey2019",
  "controller": "did:example:pqrstuvwxyz0987654321",
  "bitcoinAddress": "1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2"
}]

Failed Requirements

Rather than focusing on the requirements that are met, the list below focuses on the requirements that each option fails to achieve.

• Option A fails requirements 2, 3, 4, 6, 7, and 8.
• Option B fails requirements 5, 9, and 10.
• Option C fails requirement 9 and 10.

The use cases, requirements, and options are debatable and incomplete. This is a first attempt at providing a holistic analysis of the problem space. I'll update as more input rolls in.

[selfissued]

Thanks, @msporny for this useful analysis. In a few ways, I believe that it is inaccurate and incomplete. The way that it is most incomplete, as I see it, is that it missing the following requirement.

Requirements

11. An implementation that can check signatures/proofs for a particular key type will be guaranteed to keep working for all future instances of that key type, because alternate representations requiring additional code to translate key formats will not be introduced. (I view this code stability requirement as being one of the most important.)

Future-proofing of this kind is the whole point of standardization.

Failed Requirements

The Failed Requirements analysis in @msporny 's characterization is inaccurate in a few respects.

Option A (JWK representation) does meet these requirements:

2. Metadata can be associated with public keys. JWKs are JSON and new fields can be defined by applications to represent metadata about the public key. "use", "key_ops", "alg", and "exp" are examples of such metadata already in use. The IANA JSON Web Key Parameters registry https://www.iana.org/assignments/jose/jose.xhtml#web-key-parameters can be used by any specification to extend the set of JWK metadata.

3. JWK does minimize the encoding burden because at most one encoding is required for implementations, rather than a potentially ever-increasing multitude of equivalent encodings.

4. JWK does minimize the decoding burden because at most one decoding is required for implementations, rather than a potentially ever-increasing multitude of equivalent encodings.

5. JWK does align with encodings used by other W3C standards, such as WebCrypto. Note that WebAuthn uses CBOR encodings because messages are sent to FIDO 2 authenticators in binary, where DIDs have no requirement to support binary. We should choose JSON for all representations.

6. JWK can be used to express other proof techniques by defining new key types and key parameters, which can be done by any publicly-available specifications.

7. As for 7, JWK can be extended to support multi-signature methods by defining new JWK fields.

8. Only Option A (JWK representation) meets the code stability requirement. Both B and C require code to be updated on an ongoing basis to support new syntaxes for doing exactly the same thing - resulting in ongoing and persistent interop problems.

[OR13]

I think the point about every DID consumer already being forced to use JSON is valuable. JWK is the best way to represent keys in JSON IMO, if we had to pick only one representation, I would choose JWK. I also think that harmonizing JSON-LD and JOSE is already underway in a number of other areas, and deciding to support publicKeyJwk format here would help with that a lot.

[msporny]

Future-proofing of this kind is the whole point of standardization.

Agreed, added your requirement 11 to the original list.

[msporny]

Just a few observations, because this is useful to tease out a shared understanding of the options on the table.

2. Metadata can be associated with public keys. JWKs are JSON and new fields can be defined by applications to represent metadata about the public key. "use", "key_ops", "alg", and "exp" are examples of such metadata already in use. The IANA JSON Web Key Parameters registry https://www.iana.org/assignments/jose/jose.xhtml#web-key-parameters can be used by any specification to extend the set of JWK metadata.

While this is true, "new fields can be defined by applications" and "can be used by any specification" typically involves a non-trivial centralized standardization process vs. a more decentralized extensibility model (such as that employed by the VC and DID specs). For example, I expect that a request to add foobar to the registry would be denied even though our application absolutely requires that field on a JWK?

3. JWK does minimize the encoding burden because at most one encoding is required for implementations, rather than a potentially ever-increasing multitude of equivalent encodings.

If you're doing RSA and you don't do PEM, then you need two data formats... PEM /and/ JWK... and conversion libraries between the two. If you're doing ECDSA (Bitcoin / Ethereum) secp256k1, you need JWK and hex encoding or Base58Check encoding. If you're doing Ed25519 you need JWK and conversion to native format (which is almost always just a raw byte string).

One could argue that JWK increases the encoding/decoding burden based on what native libraries have implemented today.

4. JWK does minimize the decoding burden because at most one decoding is required for implementations, rather than a potentially ever-increasing multitude of equivalent encodings.

In theory, this should be true... but in reality, it's questionable... openssl takes PEM format (and still doesn't support JWK, AFAIK), Bitcoin/Ethereum libs don't regularly take JWK -- preferring their native format, same for a variety of Ed25519 implementations...

If JWK was a format widely recognized by libraries such as openssl, bitcoin libs, ethereum libs, ed25519 libs, etc... then there would be a strong argument to standardize on only it immediately. If JWK becomes that format in the future, there would be a strong argument to standardize on it then. That WebAuthn didn't pick JWK and went with COSE instead should be an indicator that, while it would be wonderful for the world to standardize on one key expression format, in reality, different applications make different encoding choices and the spec should be built in a way that matches that reality.

To put it another way, if we could get the openssl, Bitcoin, Ethereum, and WebAuthn communities to support JWK natively, we'd be in a much better place.

6. JWK does align with encodings used by other W3C standards, such as WebCrypto. Note that WebAuthn uses CBOR encodings because messages are sent to FIDO 2 authenticators in binary, where DIDs have no requirement to support binary. We should choose JSON for all representations.

DID Documents do have a requirements 3, 4, and 6... which are requirements to support binary, because that's what the applications expose and we don't want to go through an unnecessary translation layer. If the WebAuthn spec kicks out a binary blob for a public key, we should be able to base-encode it and stuff it right into a DID Document, zero translation necessary.

7. JWK can be used to express other proof techniques by defining new key types and key parameters, which can be done by any publicly-available specifications.

Item 7 is more about proof of work, proof of elapsed time, invocation proofs and a variety of things that have little or nothing to do with public keys. I think it would be a stretch to state that a proof of work should be encoded as a JWK.

8. As for 7, JWK can be extended to support multi-signature methods by defining new JWK fields.

This is a fair point.

Only Option A (JWK representation) meets the code stability requirement. Both B and C require code to be updated on an ongoing basis to support new syntaxes for doing exactly the same thing - resulting in ongoing and persistent interop problems.

Interesting, don't know why this is being asserted.

An implementation checks the type field for the key... the type field is tied to a year-stamped spec (such as Ed25519VerificationKey2019) with well known encoding parameters (because they're defined in the spec). Once the spec is "done" (based on the editors/authors of the spec), the parameters never change, so implementations don't need to be updated. This is true for all options, so don't see this as a differentiator, but perhaps I'm missing something?

[OR13]

Example of a resolver middleware key conversion code @tplooker mentioned on the call: https://github.com/decentralized-identity/interoperability/blob/982a09d4ad62a2be59b0f0da7d7d2ad8da1f9d9b/projects/cli-demo/src/resolver.js#L23

I expect we will see a lot of this kind of thing on clients, or an abundance of trusted resolvers who will do it for you....

the most common conversion code will likely be from X to jwk or pem.

[kdenhartog]

In regards to size discussion, I was doing a character comparison between JWK and Base58PublicKey specifically.

In the example I mentioned on the call, I was considering the comparison between a pure JWK

{
  "kty" : "OKP",
  "crv" : "Ed25519",
  "x"   : "11qYAYKxCrfVS_7TyWQHOg7hcvPapiMlrwIaaPcHURo",
  "d"   : "nWGxne_9WmC6hEr0kuwsxERJxWl7MmkZcDusAxyuf2A",
  "kid" : "FdFYFzERwC2uCBB46pZQi4GG85LujR8obt-KWRBICVQ"
}

and the raw key data (which @tplooker pointed out to me afterward is incorrect): 02b97c30de767f084ce3080168ee293053ba33b235d7116a3263d29f1450936b71

Rather the comparison should be between the two objects which give context about the keys. For example, the JWK should be compared to the object below because the type, as well as the publicMultikey field, provide context around the encoding scheme and type of key which makes this a more fair comparison. See a proper object below:

{
  "id": "#keys-4",
  "type": "Ed25519VerificationKey2018",
  "publicMultikey": "zH3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV"
}

For comparison, the JWK representation (take notes these are different keys, but both ed25519 key types) is 211 characters. The second format is 130 characters. The encoding scheme used in the second format also plays into this to a degree. (e.g. using a multikey format adds more characters than base58)

However, the point I wanted us to be aware of is that the difference in 70 characters becomes quite significant for a DID registry at scale. For example, let's assume there are 100,000,000 DIDs in a DID registry and each DID holds only 1 key. The difference in size between nodes to store all DID Docs (e.g. it's a full node storing all DID Doc data), this one implication would be a 7GB difference.

7GB doesn't sound like much of a concern right now considering storage is quite cheap. At larger scales and with more keys being used, this bloat can grow which is why I point it out. For example, 10 billion DIDs that contain only 1 key would be a difference of 700GB of data. If we say each identity has 3 keys and there are 10 billion DIDs then it becomes 2.1 terabytes of excess data had we selected a different representation.

With that in mind, my ask is that no matter the representation we choose, we're considerate of the size of the key representation. It's not a huge point now, but at an internet-scale, it could become a problem in specific implementations where nodes must replicate the entire state of the DID registry (e.g. a blockchain without pruning).

I was just trolling through the JWE spec and saw this example of an RSA key being used in RSA-OAEP in JWK format:

{"kty":"RSA",
      "n":"oahUIoWw0K0usKNuOR6H4wkf4oBUXHTxRvgb48E-BVvxkeDNjbC4he8rUW
           cJoZmds2h7M70imEVhRU5djINXtqllXI4DFqcI1DgjT9LewND8MW2Krf3S
           psk_ZkoFnilakGygTwpZ3uesH-PFABNIUYpOiN15dsQRkgr0vEhxN92i2a
           sbOenSZeyaxziK72UwxrrKoExv6kc5twXTq4h-QChLOln0_mtUZwfsRaMS
           tPs6mS6XrgxnxbWhojf663tuEQueGC-FCMfra36C9knDFGzKsNa7LZK2dj
           YgyD3JR_MB_4NUJW_TqOQtwHYbxevoJArm-L5StowjzGy-_bq6Gw",
      "e":"AQAB",
      "d":"kLdtIj6GbDks_ApCSTYQtelcNttlKiOyPzMrXHeI-yk1F7-kpDxY4-WY5N
           WV5KntaEeXS1j82E375xxhWMHXyvjYecPT9fpwR_M9gV8n9Hrh2anTpTD9
           3Dt62ypW3yDsJzBnTnrYu1iwWRgBKrEYY46qAZIrA2xAwnm2X7uGR1hghk
           qDp0Vqj3kbSCz1XyfCs6_LehBwtxHIyh8Ripy40p24moOAbgxVw3rxT_vl
           t3UVe4WO3JkJOzlpUf-KTVI2Ptgm-dARxTEtE-id-4OJr0h-K-VFs3VSnd
           VTIznSxfyrj8ILL6MG_Uv8YAu7VILSB3lOW085-4qE3DzgrTjgyQ",
      "p":"1r52Xk46c-LsfB5P442p7atdPUrxQSy4mti_tZI3Mgf2EuFVbUoDBvaRQ-
           SWxkbkmoEzL7JXroSBjSrK3YIQgYdMgyAEPTPjXv_hI2_1eTSPVZfzL0lf
           fNn03IXqWF5MDFuoUYE0hzb2vhrlN_rKrbfDIwUbTrjjgieRbwC6Cl0",
      "q":"wLb35x7hmQWZsWJmB_vle87ihgZ19S8lBEROLIsZG4ayZVe9Hi9gDVCOBm
           UDdaDYVTSNx_8Fyw1YYa9XGrGnDew00J28cRUoeBB_jKI1oma0Orv1T9aX
           IWxKwd4gvxFImOWr3QRL9KEBRzk2RatUBnmDZJTIAfwTs0g68UZHvtc",
      "dp":"ZK-YwE7diUh0qR1tR7w8WHtolDx3MZ_OTowiFvgfeQ3SiresXjm9gZ5KL
           hMXvo-uz-KUJWDxS5pFQ_M0evdo1dKiRTjVw_x4NyqyXPM5nULPkcpU827
           rnpZzAJKpdhWAgqrXGKAECQH0Xt4taznjnd_zVpAmZZq60WPMBMfKcuE",
      "dq":"Dq0gfgJ1DdFGXiLvQEZnuKEN0UUmsJBxkjydc3j4ZYdBiMRAy86x0vHCj
           ywcMlYYg4yoC4YZa9hNVcsjqA3FeiL19rk8g6Qn29Tt0cj8qqyFpz9vNDB
           UfCAiJVeESOjJDZPYHdHY8v1b-o-Z2X5tvLx-TCekf7oxyeKDUqKWjis",
      "qi":"VIMpMYbPf47dT1w_zDUXfPimsSegnMOA1zTaX7aGk_8urY6R8-ZW1FxU7
           AlWAyLWybqq6t16VFd7hQd0y6flUK4SlOydB61gwanOsXGOAOv82cHq0E3
           eL4HrtZkUuKvnPrMnsUUFlfUdybVzxyjz9JF_XyaY14ardLSjf4L_FNY"
     }

RSA is definitely an extreme case of this problem.

[iherman]

This issue was discussed in a meeting.

• No actions or resolutions
  View the transcript

  Key representation
  Daniel Burnett: https://github.com/w3c/did-spec/issues/67
  Manu Sporny: https://github.com/w3c/did-spec/issues/85
  Manu Sporny: #85 gets into whether data is application data or DID data , or …
  Michael Jones: logged #85 based on discussion with dlongley, that application data may be formatted however the app likes, while DID data should be formatted according to the standard(s) we set
  Manu Sporny: #85 is a bit of a meta-discussion, which is hard to resolve in a concall, and will make it harder to resolve #67 which is fairly focused
  … #85 is still important, just less tangible and harder to grasp
  Markus Sabadello: there’s a separate issue for “data about the DID subject” vs “data about the DID document”: https://github.com/w3c/did-spec/issues/65. this is different from issue 85
  Michael Jones: focusing on #67 for now is fine. hope is that folks will keep #85 in mind while doing so.
  Manu Sporny: https://github.com/w3c/did-spec/issues/67#issuecomment-542480584
  Manu Sporny: [ summarizes #67 ] …
  Kyle Den Hartog: one concern is that data size will continue to grow, and that may pose a problem
  Michael Jones: base* encoding does expand the size of things; that’s a valid concern.
  … most important thing is raised as Requirement 11, about future proofing
  … secondarily, JWK proponent (and author, so somewhat biased)
  Dave Longley: regretful counter is that we have no control over formats used in the wild
  … point of expressing your key is to let apps in the wild use your key, and many apps don’t consume JWK, so requiring that all keys are expressed in JWK imposes burden on apps now using openssl or nodejs or others
  … would be fantastic if a single representation worked for all, but there are many tools already using multiple representations, so this is hard to force after-that-fact
  … forcing JWK would slow adoption of DIDs and DID Methods
  Drummond Reed: Mike, what’s your thoughts on Dave’s point?
  Kyle Den Hartog: to clarify about size – it’s not about encoding; it’s about the characters required to express the base mathematics
  Kyle Den Hartog: yeah, I can add them
  Michael Jones: [ citation/example requested ]
  … dlongley is right that any chosen format would require translation be implemented (once). idea is that once that translation is written, it’s done
  Tobias Looker: DID resolver is going to be required in any case, let this handle the heavy lifting (the representation translation) for the app
  Markus Sabadello: remember that we’re separating abstract content from concrete syntax
  Manu Sporny: we’ll certainly talk about this again next time. the question is not “JWK or something-else”, it’s “something and JWK” – what is the something, that is capable of handling PEM or PEM in CBOR or ….?
  Kyle Den Hartog: @manu thank you for that clarification. In that point, I’m less opinionated about this.
  Daniel Burnett: Thanks all. Will continue this discussion next week
  Dave Longley: options: 1. support ONLY JWK, 2. support JWK and other popular formats
  Manu Sporny: question is whether to support only JWK, or JWK-plus-others
  Manu Sporny: +1
  Manu Sporny: That’s what the group has to decide.
  Ted Thibodeau Jr: fifth time’s the charm!
  

[selfissued]

Responding to @kdenhartog 's comments about key representation size, what matters for DID documents is the size of the public key representation - not the private key. Yes, RSA private keys have lots of fields (to enable efficient Chinese Remainder Theorem computations), but RSA public keys have only two fields: "n", and "e". For a 2K key, "n" has 342 characters in JWK and "e" has 4.

Likewise, the private key field "d" shown in your example above won't be present in an EC public key representation and the "kid" is optional.

We should do apples-to-apples size comparisons by comparing unadorned public key representations.

[peacekeeper]

@kdenhartog and @selfissued how a certain DID method/registry chooses to represent keys or other things internally is not too relevant for how keys are represented in the DID document. In many of the DID methods we have today there is some kind of transformation happening anyway; that's what DID resolution is for. Yes some DID methods use the DID document format internally (e.g. Veres One), but many of them don't.

[Update.. I changed "most" to "many".. in this comment, since several DID methods do use the DID document format internally]

[kdenhartog]

@selfissued brought up the point at the end of the call that Multibase has many of the issues around not selecting a particular format leading to n number of conversions needing to be implemented. From my point of view, I see the point he's making around it not solving the conversion problem for implementors. Particularly, when suggesting multi-base (I'm by no means married to that solution nor have I implemented with it yet even) what I'm looking for is a concise representation of keys, so that what we decide at the DID Document layer becomes a common enough solution that it permeates quite commonly into the application layer. ( issue #85 )

From an implementation standpoint, if I could have my cake and eat it too, it would be the ability to have a concise representation of the key which MUST be self-describing (e.g. I don't have to know another developer is using ed25519 keys because I talked with them or read documents) and I would like it to be representable in a concise format. Right now, I would say that CWKs meet both capabilities. The difficulty then comes upstream because I either have to convert all messages to CBOR at the application layer or doing conversions from CBOR to JSON as a compromise. It also only helps for a JSON serialization format but would prove much less useful if someone wanted to use YAML or XML from what I can tell.

I'd also like to say that if I had to prioritize my asks, having to implement 1 or 2 formats would be far more preferable than to be concise. However, if we get conciseness, I believe this may help resolve issue 85 from my point of view.

[iherman]

This issue was discussed in a meeting.

• No actions or resolutions
  View the transcript

  Key representation (Issue #67)
  Daniel Burnett: https://github.com/w3c/did-spec/issues/67
  Daniel Burnett: This is the primary issue; other issues are related to it.
  Manu Sporny: We’ve had the discussion about key representation before and coming back to it with new information.
  … I made an attempt to summarize what this is about.
  … We have some use cases and requirements that hadn’t really been summarized before, so this is what my comment is trying to do.
  … We’re trying to represent cryptographic information (together with other information) in DID documents, so it can be consumed by applications.
  … Three categories of applications:
  … 1. Three categories of applications: 1. Outside our scope (not our business to define key formats, but we still want to encourage interop)
  … 2. Keys used by the DID registry (internally e.g. to control the DID document). This also includes things other than keys
  … 3. Applications that are somehow in the middle. E.g. authentication. These same mechanisms may be used by the DID registry and also outside.
  … I think there’s a lot of the back and forth with different objectives.
  … One argument: Everybody standardizes on one key format, this simplifies implementations.
  … Other argument: It would be nice to have one representation, but the reality is that we don’t. There are existing implementations that have considered this but decided not to pick a single format.
  … So today we have multiple different representations of keys and we have to acknowledge that, and transformations between formats are happening
  Michael Jones: A friend in the industry said: ‘Making choices that don’t matter. To get interoperability you have to make a choice. It doesn’t matter so much what you choose, but it’s important that you do choose.’
  … I look at what’s been happening so far, and I think nobody makes any choices. Everything seems loose, the union of what everyone wants, but in the end this doesn’t serve anyone
  Kyle Den Hartog: Question for markus_sabadello about your comment in the Github issue.
  Markus Sabadello: The question you’re asking about the comment in IRC… I was trying to argue… size of different key representations, issue for certain did methods, registries, size of representation too large… whatever representation or format we use, is not necessarily what is stored in the registry. The way a lot of DID Methods work, they don’t store DID Document internally… it is a result of resolution process.
  … Bitcoin, Ethereum, they don’t actually store DID Document on the ledger, whatever we agree on, doesn’t mean, doesn’t impact how DID Documents get implemented internally. DID Document may or may not use DID Document internally.
  Kyle Den Hartog: Thanks Markus, that makes me feel like my point is more a moot point and is resolved from the registry perspective. At the application layer this has larger implications.
  Manu Sporny: I wanted to focus on real-world repercussions of this discussion. I agree with selfissued that DID spec writers are not forcing one way of doing it
  … There was a heated discussion on forcing one key format vs. allowing multiple formats
  … Both choices could result in more complexity for implementors
  … Many implementations don’t use JWK today and have chosen not to use it. They use e.g. binary encodings of keys. I think that’s terrible, but that’s how it is.
  … E.g. openssl is not going to support JWK, therefore if DID document requires JWK, transformations have to be implemented.
  … Example of Ed25519: Most people just use raw binary encoding
  … In Bitcoin and Ethereum communities, people use different formats even though the key types are the same
  … This is how the ecosystem has developed. Attempts for people to agree on a single format have failed
  … Let’s support JWK and keep trying to get people to use the same format. But also recognize that this will increase, not decrease, implementation burden
  Michael Jones: Clarification question: We’re only taking about public key representations, right?
  Manu Sporny: Yes
  Michael Jones: About your complexity argument: If you force a single format, translation has only ever to happen once (between other format, and the single support format)
  … In the current ecosystem however, transformation is more complex since every format has to be transformed to every other. This gets worse with additional formats
  … Therefore, making one choice, the overall translation problem gets easier
  … We could say that for known key types (RSA, EC) we could choose JWK and one or two other common formats, but also state that we will accept no other
  … I don’t like that, but it’s better than being completely open-ended
  Daniel Burnett: I think standards is about writing down where you can get agreement. Let other things wait
  … Even if you have to leave some things open (they may become clearer over time)
  Manu Sporny: Agree with selfissued, trying to push JWK isn’t going to result in end state selfissued wants. I also want everyone to use the same key format. But in reality I don’t think we will be able to change that.
  … Agree with selfissued to do the next-best things: E.g. accept 2 most popular formats but then don’t allow others. Aggressively narrow the list, stop proliferation
  Kyle Den Hartog: I was confused by selfissued ‘s comments about RSA and EC key types. But I thought we’re talking about different serialization formats, not key types.
  … +1 to manu (which was +1 to selfissued)
  … I think we should restrict formats, not key types (e.g. quantum crypto will become important)
  Daniel Burnett: Perhaps we schedule an additional dedicated call for this topic
  Kyle Den Hartog: +1 to that
  Michael Jones: My point was that for a given key type (e.g. RSA) we should choose 1 (or maybe 2) supported representations for this key type.
  Manu Sporny: Agreeing more and more with selfissued. Another wrinkle is something markus_sabadello highlighted on the last call: Don’t assume that DID documents will always be encoded as JSON.
  … Like in the VC spec, we separate the abstract data model from the concrete format. E.g. you could represent a DID document in a binary format like CBOR.
  … Therefore we should keep this in mind when talking about public key formats.
  … I think JWK can also be represented in CBOR, but let’s keep this in mind
  Michael Jones: This is a fine point. I find multibase particularly offensive, since that format itself refuses to make a choice;
  Daniel Burnett: We don’t have enough time to resolve this, but I feel we’re getting to something. It may be best to have a dedicated call with those who are interested.
  Manu Sporny: You can do multibase, but force ONE encoding… you do that because you can’t predict 10 years in the future… who thought that base58 would be a thing… but it is
  Kyle Den Hartog: I’d be interested to join this
  Manu Sporny: We need more than selfissued and kdenhartog and the editors to resolve this. We have to make sure we get consensus of the whole WG.
  Manu Sporny: Summary of where we are right now…
  Summary: It sounds like we’re closing in on a path forward, which is to allow multiple types, but limit to perhaps 2 mechanisms for each key type… for example, for RSA, PEM encoding and JWK encoding… While this doesn’t get us down to one key format, it does limit an explosion of key formats. We will need to have further calls to ensure that we’re taking all use cases and requirements into account, and those calls will be scheduled by the Chairs and Staff. We need to make sure that we have more than a handful of people providing input (Manu Sporny)
  

[awoie]

I'm in favor of having a few different formats. I believe it is much easier to implement the conversion at the application level. If an application requires a specific format, then it will be less effort to just implement the conversion from a few mandatory formats to the format the application needs. I assume in many cases that won't be required at all. One application where this might be relevant is verifiable credentials/ presentations where a lot of different DID methods might be used across issuers and verifiers. Of course, that won't solve the problem as a whole as once the key was parsed, the application still needs to implement the dedicated handling for the different verification methods. So, I'm happy with the current direction of WG discussion.

I don't think it matters that JWK is potentially bigger as the DID Document often does not get stored as it is -- e.g., on a ledger. It could matter to IOT devices that have limited memory though.

@selfissued If we can find a way to use JWK to encode ethereumAddress, bitcoin addresses and other non-public key type verification methods, I won't be opposed to that proposal.

[selfissued]

Per https://github.com/w3c/did-core/issues/55#issuecomment-549385979, it sounds like an ethereumAddress is actually a cryptographically-derived Key ID - not a key. JWK secp256k1 keys (bitcoin addresses) are defined in https://tools.ietf.org/html/draft-ietf-cose-webauthn-algorithms . The JWK key types are extensible, including from non-IETF specs, using the IANA JOSE registry at https://www.iana.org/assignments/jose/jose.xhtml.

[OR13]

I propose we open a PR which mandates that publicKeyJwk MUST be supported.

Next, we should answer the question why do we require another key format?

Sure we could also support publicKeyPem, but do we need to?

https://tools.ietf.org/html/rfc8037 https://tools.ietf.org/html/draft-ietf-cose-webauthn-algorithms-01

Ed25519, secp256k1, both supported.

Anything that is consuming a DID Document is using some form of resolver... A resolver, can be wrapped to provide key conversion... transforms may eventually become standardized, see:

https://github.com/decentralized-identity/papers/issues/23#issuecomment-550428699

The question is how is key conversion related to the did-core data model?

https://w3c.github.io/did-core/#contexts

"DID method specifications MAY define their own JSON-LD contexts. However it is NOT RECOMMENDED to define a new context unless necessary to properly implement the method. Method-specific contexts MUST NOT override the terms defined in the generic DID context."

If a DID Method wanted to support other key formats, or needed to, it can 100% go ahead and do that... today!

It might help us keep the "generic DID context" cleaner by establishing a clear distinction between MUST support, and optional support... its hard not to conflict with a generic did context that attempts to define lots of optional support for key formats, capabilities, authorization, etc...

Its much easier to update a context you control then it is to get ethereumAddress, publicKeyPgp, publicKeyJwk added to a context you don't... I think we should establish a pattern for did implementers to use the features of contexts, and if they don't want to use JSON-LD, to understand how to extend the "generic did context" without using JSON-LD...

We can in theory have a single key format that is MUST support, I advocate for choosing JWK, and immediately demonstrating how we can use the other extensibility features of the did spec (contexts), to support publicKeyPem.

How much code needs to be changed to keep what we have today and do the following:

"@context": ["https://www.w3.org/ns/did/v1", "https://did-method.example.com/ns/did-pem/v1"]

Why not define a context that just defines, PEM, Multibase, Ethereum, etc... and let a DID Method Implementer pick the contexts that they want... while requiring them to support interoperability with JWK... as we currently require them to conform to the DID Spec.

@dlongley am I misunderstanding the extensibility features of context here?

The whole reason I originally encountered this key representation issue was because of contexts, maybe they are the solution?

[csuwildcat]

I want to echo what @OR13 said about having a JWK as a base format for descriptors, given I have not found any significant reason why we can't represent keys within the bounds of JWK, with the addition of new key types, if necessary. Method resolver code should be able to transform from whatever a Method uses internally to the base format, and such a simplification would help make this aspect of the spec easier to deal with.

[selfissued]

My comment https://github.com/w3c/did-core/pull/100#pullrequestreview-314703524 requests that we document the agreement to support the JWK representation for all public key types and that any additional representations would be an exception, as agreed to on the 6-Nov-19 special call. That will take us a long way towards the interop and simplicity goals.

I do believe that @OR13 is asking the right question when he asks "Next, we should answer the question why do we require another key format?". We should try to cover that during the next special call.

[msporny]

@selfissued wrote:

My comment #100 (review) requests that we document the agreement to support the JWK representation for all public key types and that any additional representations would be an exception, as agreed to on the 6-Nov-19 special call.

That is defintiely not what I thought we had consensus on. I agree that we were trying to support JWK across all key formats. I didn't think that the group agreed that other representations would be exceptions (as many key formats deployed today do not use JWK, even though some variation of JWK has been available for use for the past 8 years or so). More below...

@OR13 wrote:

I propose we open a PR which mandates that publicKeyJwk MUST be supported.

I think that's what the current PRs attempt to do? What am I missing?

@selfissued wrote:

Next, we should answer the question why do we require another key format? Sure we could also support publicKeyPem, but do we need to?

I think the answer is: there is no one supported key encoding format today. There have been attempts to get alignment (JWK), but those attempts have not resulted in what we'd like to see (which is one supported key encoding format across the entire public key expression ecosystem).

I think @kdenhartog's point here cuts to the heart of the matter:

https://github.com/w3c/did-core/pull/102/files#r344938209

@kdenhartog wrote:

I'm not sure if anyone is currently using JWKs as a representation right now for ed25519 keys, (legacy implementations appear to use base58) but I read this as implementers of future DID registries are encouraged to use JWKs rather than base58.

In other words, by mandating one key format (that implementers had an opportunity to use, but didn't), we're not reflecting reality... and specs should reflect reality. If we don't do that, we have an academic ivory tower spec... not a pragmatic one (and the latter specs are the ones that get deployed). My understanding is that we were headed towards supporting multiple key formats. The latest arguments seem to be backpedaling to one key format (yes, that would be ideal... but reality states that this has not been happening over the past decade). Some data points to consider:

1. RSA libraries almost universally understand PEM encoding for public keys.
2. Ed25519 libraries almost universally understand raw bytes for public keys (with base58 encoding).
3. secp256k1 libraries almost universally understand raw bytes for public keys (with base58 or hex encoding).

To mandate one key format will increase implementation burden, not reduce it.

If we are going to mandate the use of JWK, we're going to have to see if implementers are actually going to implement that solution. I suggest that, given the current trajectory of the conversation, that we get some real data from as many of the cryptographic library implementations as possible. Which libraries support which formats today, and use that to guide the discussion. Bonus points if we can get implementers of those libraries to solidly commit to supporting a single key format (like JWK).

We need a reality check, so let's do some science and get data from implementations and, if we can, implementers. I suggest that we:

1. Do a survey of cryptography libraries to see what public key formats they support.
2. Do a survey of DID Methods to see what public key formats they plan to support (and see if they have a preference).
3. See if we can get 1 and 2 above to commit to a single key format (JWK being the likely candidate) across all key formats.

[msporny]

@OR13 wrote:

@dlongley am I misunderstanding the extensibility features of context here?

You are understanding the extensibility features correctly. The issue we've had in the past is that there is a small subset of the community that doesn't want to use those features. The more @context values there can be, the more complex the non-JSON-LD processors need to become. Note that this isn't an issue for JSON-LD processors, but is an issue for pure JSON ones. The burden isn't too great, unless you're talking about having a stack of 5+ contexts, and then you have to possibly do combinatorial matching on a subset of the @context array if you choose to not use a JSON-LD processor. It's doable, but this is why we're trying to get some of these values into the base context... so folks don't have to do that sort of stuff in non-JSON-LD environments.

Our assumption is that you're going to have the base DID context and then a DID Method specific context, and that's it:

"@context": ["https://www.w3.org/.../did/v1", "https://my-did-method.com/.../mydid/v1"]

The whole reason I originally encountered this key representation issue was because of contexts, maybe they are the solution?

Yes, we could solve the issue like that, but what I heard @selfissued say on the call last week is that he would object to that approach. He would not like there to be other possibilities outside what the DID Core spec allows, even if a DID Method wants there to be. Did I misunderstand that requirement of yours, @selfissued?

[msporny]

Do a survey of cryptography libraries to see what public key formats they support.

I have started a spreadsheet here to gather the findings. I need help filling the spreadsheet out, everyone has edit privileges:

https://docs.google.com/spreadsheets/d/12dwUaAruKKpq3a3IfEMEMhpRhI7oM1tQnkmDbW2VGoU/edit#gid=0

Do a survey of DID Methods to see what public key formats they plan to support (and see if they have a preference).

I have started a survey here to gather data from DID Method implementers:

https://forms.gle/Hovf3irBJ5KwgXLQ6

[selfissued]

@msporny wrote that "specs should reflect reality". That's certainly a good principle, but it's often the case before there's a standard in an area that the reality is that are many divergent implementations of essentially the same thing that differ in unnecessary ways.

A concrete example of this was signed JSON claims before the JWT standard. As you can read at https://self-issued.info/?p=1387, in 2010 there were at least four different formats for this in use by different systems. It seemed like time for a standard! We surveyed the features, created and refined a specification that would work for all their use cases, and created a standard. The reality now is all all these non-standard precursor formats are gone in favor of JWT.

The DID world is now in a similar place to where signed JSON was in 2010. There's been lots of useful prototyping and experimentation. People made different choices around things that needn't be different. There's no standard for them to coalesce around to create interoperability.

We have the opportunity now to create a DID standard that will foster interoperability, eliminating unnecessary differences. Taking the path of saying that "reality has a multiplicity of divergent representations - let's leave it that way" would be doing everyone a disservice, and would fail to utlitize the power of standards to create an interoperable reality for the future.

[msporny]

From the 2019-11-13 telecon:

Proposed resolution: Standardize on JWK as the only public key format that MUST be supported. Michael Lodder: -1 Yancy Ribbens: -1 Orie Steele: -1 Dave Longley: -1 Jonathan Holt: -1 Oliver Terbu: -1 Michael Jones: +1 Proposal failed to gain consensus.

Proposed resolution: Standardize on JWK and PEM as the only two supported key formats for at least RSA, secp256k1, secp256r1, ed25519, Curve25519. At least ONE format MUST be supported. Michael Lodder: -1 Dave Longley: -1 Yancy Ribbens: -1 Michael Jones: -1 Orie Steele: +1 Jonathan Holt: -1 Proposal failed to gain consensus.

Proposed resolution: Standardize on JWK (FormatA) and a per key type format as the only two supported key formats for at least RSA, secp256k1, secp256r1, ed25519, Curve25519. At least TWO formats MUST be supported. Michael Jones: +1 Orie Steele: -1 Dave Longley: -1 Michael Lodder: -1 Yancy Ribbens: -1 Proposal failed to achieve consensus

Proposed resolution: Standardize on JWK (FormatA) and a per key type format as the only two supported key formats for at least RSA, secp256k1, secp256r1, ed25519, Curve25519. At least ONE format MUST be supported. Orie Steele: +1 Michael Lodder: +1 Markus Sabadello: +1 Dave Longley: +1 to OR Yancy Ribbens: +1 Brent Zundel: +1 Oliver Terbu: 0 Jonathan Holt: -1 Michael Jones: 0 Proposal achieved consensus.

Proposed resolution: There will be a small fixed set of key representations for DID documents described in the spec. Michael Jones: +1 Orie Steele: +1 Dave Longley: +1 plus – you can extend to add more but not for the same key type Tobias Looker: +1 Markus Sabadello: +1 Oliver Terbu: +1 Michael Lodder: +1 Brent Zundel: +1 Yancy Ribbens: +1 Proposal achieved consensus.

Note: I have removed duplicate votes from the same organization to ensure that the votes weren't being packed by any particular organization, but only after confirming that all people from each organization voted the exact same way. Every proposal above was a straw poll and not a binding vote by the organization. The purpose of the exercise was to get a temperature of where the discussion was and not to create a binding decision by the group.

[TallTed]

Note: I have removed duplicate votes from the same organization to ensure that the votes weren't being packed by any particular organization, but only after confirming that all people from each organization voted the exact same way.

In future, I suggest identifying such compressed votes by Org name, perhaps with parentheticals of the individual member names.

That said, my experience of practice in WGs has generally been that while individuals present both their personal and corporate positions, which are often but not always in agreement -- WG votes are usually counted as one person one vote, which votes are usually based on their personal positions, because WG members are expected to be acting primarily as subject matter experts and only secondarily as corporate representatives.

[msporny]

In future, I suggest identifying such compressed votes by Org name, perhaps with parentheticals of the individual member names.

Will do, note that the minutes had the pre-compressed straw poll votes. We can always refer back to the original record for the pre-compressed straw poll votes.

That said, my experience of practice in WGs has generally been that while individuals present both their personal and corporate positions, which are often but not always in agreement -- WG votes are usually counted as one person one vote, which votes are usually based on their personal positions, because WG members are expected to be acting primarily as subject matter experts and only secondarily as corporate representatives.

Yes, in an ideal world (and I wish it worked that way)... but I've also seen companies pack the room with their own representatives, or representatives that never show up except for a vote... so, felt like compressing the straw poll votes to one org was safer. I did this primarily because there were four people from Digital Bazaar (my organization) voting and I could see others in the WG objecting to the findings based on an insistence that we were "packing the room"... we just happen to have folks from our company that are regulars and want to participate (and have differing opinions on where things should go... I'll cite Dave Longley, Dmitri, and I arguing against each others points from time to time on the calls... same org, very different opinions).

In any case, I felt it safer to compress the straw poll votes to avoid the whole "you were packing the room, it doesn't count" argument that some people use when things don't go their way.

[peacekeeper]

The idea has come up that public key representations could be automatically converted during DID resolution. Here's an experiment that can convert to JWK using a "transform-keys" matrix parameter, e.g.:

curl -X GET "https://uniresolver.io/1.0/identifiers/did:sov:WRfXPg8dantKVubE3HX8pw;transform-keys=jwk"
curl -X GET "https://uniresolver.io/1.0/identifiers/did:btcr:xz35-jznz-q6mr-7q6;transform-keys=jwk"
curl -X GET "https://uniresolver.io/1.0/identifiers/did:key:z6Mkfriq1MqLBoPWecGoDLjguo1sB9brj6wT3qZ5BxkKpuP6;transform-keys=jwk"

For some additional background, see https://hackmd.io/XmL-Bjh5TdqV4fj6nwdPEQ

Note: I'm not actually proposing to introduce this matrix parameter at this point. It may actually be better to do something like this with a resolver input option, rather than a DID URL matrix parameter. Just wanted to add this as input to the discussion.

[tplooker]

To clarify the discussion on the call today, I believe the current state of the PR's in regards to this issue states that a particular public key type has a set of valid representations within a did document, where one of these MUST be used by a piece of software that is creating the representation of a particular public key.

I think the point you wished to clarify @selfissued is that if there are multiple public key type representations for a given public key type, then the software consuming the did document representation MUST support all formats valid for a particular key type to guarantee they can consume the did document?

[selfissued]

Yes. Accepting all legal inputs is a proven way to facilitate interoperability.

[OR13]

^ I think thats pretty much it.

For all DID Methods, it is valid to return a DID Doc where all keys are in JWK format.

DID Methods may choose not to support RSA keys, or not to use JWK, thats up to the Method implementer.

If a DID Method decides not to use JWK, it MUST select a key format from the list of approved key formats for each key type specified in the spec, and that list can't keep growing...

^ This is likely to be the greatest area of contention.

I think there are some technical issues underpinning positions members of the community are taking.

There are DID Documents which contain base58 and hex encoded keys which are signed and therefore, the key format is basically chosen by the client, and can't be changed.

Method implementers that have this constraint have no option but to advocate for the key format that is already baked in, so they will always need at least the format they have chosen that is not JWK to be valid for the key type according to the did-core spec.

There are 2 ways of handling this.

A. The DID Method creates its own context and adds support for the custom format to its DID Method key formats that are required by this method.

B. The DID Spec context defines support for these key formats, and allows for their use by everyone.

Im in favor of keeping the did core context as thin as possible, and I'm worried that specific method implementer decisions (or ledger specific formats) are polluting the core spec context, when they really belong in the method context...

However, its also bad to have the method context define functionality that is core to interoperability...

The current spec notes:

The following is a non-exhaustive list of public key properties used by the community: publicKeyPem, publicKeyJwk, publicKeyHex, publicKeyBase64, publicKeyBase58, publicKeyMultibase, ethereumAddress.

IMO at this point, our job is to decide which of these are getting eliminated from the did-core context (they can still be added back to the method specific context).

These values are mostly method specific and IMO should be excluded from the core context (you can still support them in your method context, its your DID Method after all):

publicKeyHex, ethereumAddress, publicKeyBase58

These values are candidates for inclusion in the core context IMO:

publicKeyPem, publicKeyMultibase, publicKeyBase64

I'm not sure how valuable RSA support is these days, I'll make the following proposals:

Prop OS1: If keys are not represented as JWK, they must be represented as publicKeyMultibase.

Prop OS2: If keys are not represented as JWK, they must be represented as publicKeyMultibase, or publicKeyPem.

Prop OS3: If keys are not represented as JWK, they must be represented as publicKeyMultibase, or publicKeyPem, or publicKeyBase58.

I do think that its pretty unreasonable to choose to support something that does not even have an RFC / spec at the did-core context level... If you want to do that, I think we should advise method implementers to use their own context to do so.

Here are the IETF specs relevant to this discussion:

• https://tools.ietf.org/id/draft-multiformats-multibase-01.html
• https://tools.ietf.org/wg/pem/
• https://tools.ietf.org/html/rfc4648#section-5

[kdenhartog]

Just so we don't miss it, should we be adding a definition by NIST P curves and how they should be represented in the DID Document? As far as I'm aware, no one is using these curves because ledgers typically don't support them (Hyperledger Fabric is the only one I can think of that may use them), but they're the standard elliptic curves mandated by governments typically. What are other people's thoughts on adding them? I think it's obvious that we support JWKs with them, but would we need to support another format as well? If so, which type would that be?

[OR13]

I think we have support for them, by allowing all keys to be expressed with JWK.

We probably still need JSON-LD context support for every linked data signature and key for each, but thats not gated by a key format discussion, its gated by updating JSON-LD contexts.

[msporny]

PR #100 has been merged, which addresses this issue -- the spec now clearly defines which public key formats are supported. We still need to determine the base encoding format, additional encoding formats such as hex and ethereumAddress, and how id and kid are related. There are separate issues for those. Closing this issue.