Protected term redefinition `Ed25519Signature2018` when adding custom ldp-bbs 2020 contexts

When using a custom document loader to swap the unregistered BBS context (PR open https://github.com/w3c-ccg/ldp-bbs2020/pull/33) with a local copy and trying to sign a verifiable presentation, jsonld gives an error saying Protected term redefinition and term: Ed25519Signature2018

details: {
    code: 'protected term redefinition',
    context: {
      id: '@id',
      type: '@type',
      dc: 'http://purl.org/dc/terms/',
      sec: 'https://w3id.org/security#',
      xsd: 'http://www.w3.org/2001/XMLSchema#',
      EcdsaKoblitzSignature2016: 'sec:EcdsaKoblitzSignature2016',
      Ed25519Signature2018: 'sec:Ed25519Signature2018',
      EncryptedMessage: 'sec:EncryptedMessage',
      GraphSignature2012: 'sec:GraphSignature2012',
      LinkedDataSignature2015: 'sec:LinkedDataSignature2015',
      LinkedDataSignature2016: 'sec:LinkedDataSignature2016',
      CryptographicKey: 'sec:Key',
      authenticationTag: 'sec:authenticationTag',
      canonicalizationAlgorithm: 'sec:canonicalizationAlgorithm',
      cipherAlgorithm: 'sec:cipherAlgorithm',
      cipherData: 'sec:cipherData',
      cipherKey: 'sec:cipherKey',
      created: [Object],
      creator: [Object],
      digestAlgorithm: 'sec:digestAlgorithm',
      digestValue: 'sec:digestValue',
      domain: 'sec:domain',
      encryptionKey: 'sec:encryptionKey',
      expiration: [Object],
      expires: [Object],
      initializationVector: 'sec:initializationVector',
      iterationCount: 'sec:iterationCount',
      nonce: 'sec:nonce',
      normalizationAlgorithm: 'sec:normalizationAlgorithm',
      owner: [Object],
      password: 'sec:password',
      privateKey: [Object],
      privateKeyPem: 'sec:privateKeyPem',
      publicKey: [Object],
      publicKeyBase58: 'sec:publicKeyBase58',
      publicKeyPem: 'sec:publicKeyPem',
      publicKeyWif: 'sec:publicKeyWif',
      publicKeyService: [Object],
      revoked: [Object],
      salt: 'sec:salt',
      signature: 'sec:signature',
      signatureAlgorithm: 'sec:signingAlgorithm',
      signatureValue: 'sec:signatureValue'
    },
    term: 'Ed25519Signature2018'
  }
}

The bbs context has no mention of Ed25519 anywhere, so I'm not sure what's causing the issue.

BBS context:

{
  "@context": {
    "@version": 1.1,
    id: "@id",
    type: "@type",
    ldssk: "https://w3c-ccg.github.io/lds-bbsbls2020/contexts/#",
    "BbsBlsSignature2020": {
      "@id": "https://w3c-ccg.github.io/lds-bbsbls2020/contexts/#BbsBlsSignature2020",
      "@context": {
        "@version": 1.1,
        "@protected": true,
        "id": "@id",
        "type": "@type",
        "sec": "https://w3id.org/security#",
        "xsd": "http://www.w3.org/2001/XMLSchema#",
        "challenge": "sec:challenge",
        "created": {
          "@id": "http://purl.org/dc/terms/created",
          "@type": "xsd:dateTime"
        },
        "domain": "sec:domain",
        "signature": "sec:signature",
        "nonce": "sec:nonce",
        "proofPurpose": {
          "@id": "sec:proofPurpose",
          "@type": "@vocab",
          "@context": {
            "@version": 1.1,
            "@protected": true,
            "id": "@id",
            "type": "@type",
            "sec": "https://w3id.org/security#",
            "assertionMethod": {
              "@id": "sec:assertionMethod",
              "@type": "@id",
              "@container": "@set"
            },
            "authentication": {
              "@id": "sec:authenticationMethod",
              "@type": "@id",
              "@container": "@set"
            }
          }
        },
        "verificationMethod": {
          "@id": "sec:verificationMethod",
          "@type": "@id"
        }
      }
    },
    "BbsBlsSignatureProof2020": {
      "@id": "https://w3c-ccg.github.io/lds-bbsbls2020/contexts/#BbsBlsSignatureProof2020",
      "@context": {
        "@version": 1.1,
        "@protected": true,
        "id": "@id",
        "type": "@type",
        "sec": "https://w3id.org/security#",
        "xsd": "http://www.w3.org/2001/XMLSchema#",
        "challenge": "sec:challenge",
        "created": {
          "@id": "http://purl.org/dc/terms/created",
          "@type": "xsd:dateTime"
        },
        "domain": "sec:domain",
        "nonce": "sec:nonce",
        "proofPurpose": {
          "@id": "sec:proofPurpose",
          "@type": "@vocab",
          "@context": {
            "@version": 1.1,
            "@protected": true,
            "id": "@id",
            "type": "@type",
            "sec": "https://w3id.org/security#",
            "assertionMethod": {
              "@id": "sec:assertionMethod",
              "@type": "@id",
              "@container": "@set"
            },
            "authentication": {
              "@id": "sec:authenticationMethod",
              "@type": "@id",
              "@container": "@set"
            }
          }
        },
        "proofValue": "sec:proofValue",
        "verificationMethod": {
          "@id": "sec:verificationMethod",
          "@type": "@id"
        }
      }
    },
    Bls12381G2Key2020: "ldssk:Bls12381G2Key2020",
  }

REPL for error - https://repl.it/@HardikKumar/BBS-Signatures-VP-Signatures

cc @tplooker

The "Unsigned Presentation" in your logging output contains a proof that embeds a @context (that points to the security/v2 context) that redefines the Ed25519Signature term in a way that incompatible with the VC data model context:

      "proof": {
        "type": "BbsBlsSignatureProof2020",
        "@context": "https://w3id.org/security/v2",
        "created": "2020-07-14T20:45:15Z",
        "verificationMethod": "did:example:489398593#test",
        "proofPurpose": "assertionMethod",
        "proofValue": "AAsH/7IHSzIRNz0ONPtqDp+D73PkpZFMsC9P8n01kpA2PQum2sv9mEAktuQXkVE9Y3UI4rSSYLgq6kbE3ikBK0w8Th7aACAl3oHMeIGgcDCXkX7Q5TCiwdj+/DBOznMW9b6PO5PrbNqeZvXgeE/9qs9Bwr1iZDxT7C/s3RPU8i3JwJF9lfDXXh7MqvpnvmCuymdLiwAAAHSpyVOVSAPX5DM87MSgZT/26Sf6LiFTr0W4RPidQbVfQjjtC0Wg3+raGzwcFJl+WsMAAAACES4CUcaRFh7M5Md3Rvv3stGFqneDZ5fnmtqXkw+m1MFlZA6g8OaU5mY4aM2zVFGym4IalWQCCTAwIQyJfUSnz7KhAr0YeY+TJCnVNFe35ClWH5BZJLCtBhzJC38uQQgkYLf5mERShB1sw4TdBCBE3QAAAAIcI7MQ2n3Rx6bBh4O/M3cxZOGH3y9ApR8AqDU0AutAMUj1BGGEI2Y+6joC2xPC1IjVSWvai7I5mf29EEczyrPi",
        "nonce": "lhYISjY0vn7TYRnLuAb8W3WTTKna3WtP3of9curWD74/LdckH3zb4Em3yfq64d52mCw="
      }

So -- I presume that somewhere along the line that security context should not be getting injected into the proof.

Looks like deriveProof passes a proof object with the security context on it into the suite's deriveProof method:

https://github.com/mattrglobal/jsonld-signatures-bbs/blob/29987106344191819bac3073d913e39927183813/src/deriveProof.ts#L42-L55

And then the suite includes that proof wholesale:

https://github.com/mattrglobal/jsonld-signatures-bbs/blob/29987106344191819bac3073d913e39927183813/src/BbsBlsSignatureProof2020.ts#L176-L185

Without consideration of the document's (on which the proof is attached) context.

In jsonld-signatures, the ProofSet class handles attaching proofs like this:

https://github.com/digitalbazaar/jsonld-signatures/blob/master/lib/ProofSet.js#L107-L149

In short, if a flag (compactProof=false) is passed that says "I know what I'm doing, all of my proof terms will be defined by the top-level document's context", then the code will go ahead and simply delete the context from the proof (this is an optimization) and then attach it. Otherwise, it will use JSON-LD to expand the proof and then recompact it using the document's context, to ensure it matches before attaching it. If something similar were done, it would likely address this issue.

@dlongley thanks for the clarification, the logic you reference from jsonld-signatures-bbs is to ensure the proof has an associated context when you canonicalize it, but this in general needs to be reviewed including how to drop this on the output. To be clear are you saying because some of the terms we are using are not in the security context vocab we need to be compacting the proof always i.e using the documents context?

Currently in jsonld-signatures it appears in either case (compactProof or not) when you are verifying a proof you always attach the security context? https://github.com/digitalbazaar/jsonld-signatures/blob/master/lib/ProofSet.js#L304-L306

@tplooker,

To be clear are you saying because some of the terms we are using are not in the security context vocab we need to be compacting the proof always i.e using the documents context?

The proof is being added to a document that has some existing context -- which means that whatever terms that existing context defines must not conflict with whatever is in the proof. If one JSON-LD document is going to be inserted into another one (which is what is happening here), their contexts must be compatible.

So, how can contexts be incompatible? By default, JSON-LD allows terms to be redefined, simply by introducing a new @context containing different definitions for a given term. This is an important feature of any decentralized semantics technology. There are situations where you want more limitations, however, especially for certain security-oriented use cases. In this particular case, the document above uses the VC context, which uses the JSON-LD 1.1 @protected feature to define its terms. The @protected feature is strongly recommended for contexts that are intended to travel in security-oriented data. The @protected feature provides assurance that an @protected term can't be redefined in a way that could cause two different parties to interpret it differently.

Specifically, if someone reads the VC data model spec and implements to it, their software will interpret the terms according to the spec. They can get utility out of VCs without having to care about JSON-LD transformations (such as compacting to a different @context).

If another party, that is more familiar with JSON-LD, wants to consume data that happens to use the VC context, they can do so -- and even transform the data into an @context their software is familiar with, without having to read the VC spec. Their software will interpret the terms the same way, because the definitions are not permitted to change via the @protected feature. A JSON-LD processor will throw an error if it encounters an @context that attempts to change an @protected term's definition in a way that would cause a different interpretation from the party that read and implemented to the spec.

The @protected feature enables these unrelated consuming parties to go about their business without having to coordinate (decentralized FTW), where the only cost is to the data/@context author who must be careful not to redefine terms that would run afoul of the @protected rules (priority of constituencies FTW).

So, your software can do one of three things to ensure a valid merge.

Ensure that the @context used in proof has no term conflicts with the @context used in the document. This allows proof to be inserted into the document cleanly, even if it includes its own @context. Ensuring this could be a matter of designing the software using some a priori knowledge that can provide this assurance.
Ensure that all terms used in proof are already appropriated defined in the document's context -- which means you can just delete @context from proof after inserting it. This is essentially like the first option, except that the additional @context in proof is superfluous and should be removed.
Compact proof to the document's context before inserting it. This will guarantee compatibility at the cost of a JSON-LD compaction operation.

As noted above, jsonld-signatures can do option 2 or 3 -- and offers a flag to let the developer declare that option 2 is ok, if they have the knowledge that it is. If the developer doesn't set that flag, it will use option 3 to ensure a valid merge.

...the logic you reference from jsonld-signatures-bbs is to ensure the proof has an associated context when you canonicalize it...

By the way, make sure that your type value is defined. I didn't look closely at your code to see if that's the case. If the proof uses the security/v2 context and the type value isn't defined in that context -- then if that proof is canonized, it is not using the proper definition for your type value. It will instead use a relative URL (with no base URL) for the type value, i.e., /BbsBlsSignatureProof2020. This which will be dropped when converting to RDF. There is an outstanding issue to make it possible to throw an error when this happens (see #199).

Currently in jsonld-signatures it appears in either case (compactProof or not) when you are verifying a proof you always attach the security context?

This is essentially the reverse situation from the above. Here we're plucking a proof out of a JSON-LD document and we need to know what its @context is. You can see, via the link below, that if compactProof is set, the document will be compacted to the security context prior to plucking the proofs out, to ensure that attaching the security/v2 context will be valid:

https://github.com/digitalbazaar/jsonld-signatures/blob/master/lib/ProofSet.js#L286-L292

If that flag is set to false, then the developer is explicitly calling out that every term in proof is appropriately defined by the security context, so compacting the document to that context prior to pulling out the proof is not necessary (an optimization). However, if all terms in the proof are not defined by the security/v2 context, as would be the case here with BbsBlsSignatureProof2020, for example, compaction is required to produce a valid proof that jsonld-signatures can operate on.

Over time, new security context versions come out (version 3 is in the works) that aggregate more terms to enable these optimizations for more cases. Once version 3 is ready, it will be incorporated into a major revision release of jsonld-signatures to enable the optimization to happen when its terms are used. It's also possible that we could come up with a more clever API pattern to make this sort of optimization easier for custom suites -- so we're open to changes there that would simplify the process.

Thanks for the detailed explanation @dlongley

However, if all terms in the proof are not defined by the security/v2 context, as would be the case here with BbsBlsSignatureProof2020, for example, compaction is required to produce a valid proof that jsonld-signatures can operate on.

Just to be clear, the compacting operation shown here would have to be against a different context than V2 so that all the terms for the BbsBlsSignatureProof2020 were caught correctly? It appears all compact operations in jsonld-signatures are against a hardcoded context (v2), would it be dangerous or invalid to instead allow supplying this context through the api, so that signature suites that are not in the formal security context behave correctly?

Im also confused as to why compacting is required, is this because a JSON-LD linked data proof could be attached to a JSON-LD document that it does not feature the security context in its root context? If not then is it not equally valid to use the documents context for the compacting operation, is that not done to just ensure term-redefinition is not occurring?

@tplooker,

Just to be clear, the compacting operation shown here would have to be against a different context than V2 so that all the terms for the BbsBlsSignatureProof2020 were caught correctly?

No, applications and libraries are written against particular contexts they understand. The jsonld-signatures library is written against the security/v2 context. This means that when data comes into jsonld-signatures library, it should be compacted to that context so that the code will operate properly. Once data is in that context, the library code doesn't need to do any special transforms -- it just assumes the data is organized using the terms it understands. This is because the "compact" operation will transform all the terms in a JSON-LD document from whatever context they are presently in into the context that is passed to the compact API.

If there are any plugins to jsonld-signatures, they will receive the data in the security/v2 context. Those plugins may recompact data/use other contexts as they please, but if those plugins need to send data back into jsonld-signatures, they should send it in using the security/v2 context.

Similarly, when data leaves jsonld-signatures (is handed back to the caller of its APIs), jsonld-signatures will transform the data back into whatever context the caller is expecting. This context comes from the @context value in the input document. So, by default, jsonld-signatures will compact the proofs it adds to that @context. This can be avoided via the use of a compactProof=false flag that the caller may set if they know that this transformation wouldn't actually alter the terms that appear in the proof, thus removing a superfluous JSON-LD compact operation.

It appears all compact operations in jsonld-signatures are against a hardcoded context (v2), would it be dangerous or invalid to instead allow supplying this context through the api, so that signature suites that are not in the formal security context behave correctly?

JSON-LD compaction "round trip safe". If you start with a valid document using context A and compact it to context B -- you can go back again to context A -- provided that all of the terms used in the JSON-LD document are actually defined by whatever context it started with. If not, you will have other problems anyway -- and jsonld-signatures will throw an error in these cases. As explained above, the context that jsonld-signatures compacts to is, in fact, hard-coded to security/v2 because that's the context that the code is written against. This is how applications consuming JSON-LD work generally; they have a context against which they are written such that they can accept data in any other context, and then they can compact to their native context to use it. If they need to add anything to the data or hand it back, they can compact back to the originating context so that the caller gets data using the terms definitions they prefer.

Im also confused as to why compacting is required, is this because a JSON-LD linked data proof could be attached to a JSON-LD document that it does not feature the security context in its root context?

Compacting is required because the context being used by the caller is a parameter; from the perspective of jsonld-signatures, it is entirely up to them what they want to use. It is designed for fully decentralized use.

If not then is it not equally valid to use the documents context for the compacting operation, is that not done to just ensure term-redefinition is not occurring?

Per above, the conditional question here doesn't apply (because the context could be something else). However, when the caller knows that the terms definitions won't change, they may pass compactProof=false. Note that what matters are the specific term definitions, not a particular context (referenced via URL X or URL Y). If the term definitions that jsonld-signatures uses are compatible with whatever term definitions are provided in the context of the document -- then compactProof=false can be passed to avoid a superfluous compact call. If the caller doesn't know or isn't sure about this, they don't have to set that flag, in which case a compact operation will occur to ensure no trouble ensues.

Also, note that compacting to another context (in this case the security/v2) is not what triggers an error. If you change contexts, you may redefine terms as you please, because if the incoming terms don't match, they will be transformed into full URLs which will unambiguously differentiate them for the consuming code. The error is triggered when the JSON-LD processor notices that the document you passed in is invalid. The document itself uses contexts in a way that would redefine @protected terms internally and that's a mistake in its authorship. It's akin to having a JSON syntax error (like a trailing comma) -- but here it's a JSON-LD error instead. This has nothing to do with what the target context is that you're trying to compact to for your application so the data terms will fit its code natively. You can't even start that transformation because your document was invalid to begin with.

This issue has been resolved in jsonld-signatures-bbs where it stemmed from with this PR, propose closing it @haardikk21 ?

digitalbazaar / jsonld.js

Protected term redefinition `Ed25519Signature2018` when adding custom ldp-bbs 2020 contexts #402