search

w3c / vc-data-integrity

W3C Data Integrity Specification
https://w3c.github.io/vc-data-integrity/
Other
40 stars 18 forks source link

Remove normative dependency on Multibase and Multihash. #196

Closed msporny closed 10 months ago

msporny commented 11 months ago

This PR has been raised to address issue #191, which requested the removal of Multibase from the list of normative dependencies for the vc-data-integrity specification. I went ahead and removed Multihash as well from the list of normative dependencies as I expect the request applies to that specification as well.

Preview | Diff

msporny commented 11 months ago

This PR doesn't do the job. Putting a question mark in the references doesn't remove the references. Please remove them entirety, as agreed to at TPAC.

That is not what was agreed to at TPAC. The agreement was to make the references non-normative, which this PR does. Please review the minutes:

https://github.com/w3c/vc-data-integrity/issues/191#issuecomment-1720718132

Specifically:

Brent Zundel: appreciate that there's a feature at risk tag. is the intention, should multibase not be standardized by the time this moves from CR to REC, would the current link be moved to a non normative reference? Sebastian Crane: if that satisfies Mike I am happy to make that compromise myself and make it an informative reference. we are waiting for the spec to catch up. Sebastian Crane: (to summarize) - we are going to, in the cryptosuites, to have a specific encoding, for the spec text we are using a format compatible with multibase, so that if it becomes a standard we will use it normatively. we won't publish a normative ref if it's not ready by REC.

There was no opposition to that summary during the meeting. I do expect there to be opposition to removing all non-normative references to multiformats from the specification. If you would like to make that suggestion, please raise a PR (though I expect it to have a large number of downvotes since it weakens the specification by not pointing to useful, stable documentation that exists today in a non-normative fashion).

msporny commented 11 months ago

Sorry but I do not understand how adding a ? removes the normative dependency

It is a ReSpec shortcut: https://respec.org/docs/#normative-vs-informative-references

msporny commented 11 months ago

It was my understanding from the discussion at TPAC was that the resolution was going to be that the references were going to be removed entirely until such time as there is a multibase standard to reference.

As you can see from the quoted text above, that understanding was incorrect. The agreement was to make the reference non-normative, which this PR does.

And that it would be replaced by in-spec statements that these values are strings beginning with either "u" or "z" and followed respectively by either a base64url-encoded value or a base58btc-encoded value.

That spec text has existed in the DI spec for some time now:

https://w3c.github.io/vc-data-integrity/#resource-integrity

Promote these to be normative and ditch the reference to the individual Internet Draft.

The references to the I-Ds, which are non-normative now, are at the same level as linking to a Wikipedia article, or some content on the web, that is, they are informative in nature and helpful because they explain Multihash and Multibase in more depth than we do in the current specification. Removing access to documentation that is helpful to developers that want to learn more about how each technology works would be harmful to developers trying to understand more about each technology.

selfissued commented 11 months ago

Replacing the normative reference with an informative reference effectively leaves proofValue undefined. That's not OK, since it's a required field. This PR needs to be updated to normatively define the contents of the proofValue field.

OR13 commented 11 months ago

I think the proofValue issue can be addressed without a normative reference to specific text encodings.

Something like:

"Based on the @type and cryptoSuite used by the proof, the specification defining the crypto suite can define the semantics of proofValue, the JSON value MUST be a string."

Then site base58btc normatively as the string encoding, but just copy the alphabet used in to the spec.

TallTed commented 11 months ago

[@selfissued] This PR needs to be updated to normatively define the contents of the proofValue field

I'm sure an explicit change suggestion would be welcome, and more likely to satisfy you than most if not all alternatives, though it may not be 100% accepted by the rest of the WG, who may yet make comments or suggest additional changes thereon (e.g., if your change request includes fully removing the informative references to [[?MULTIBASE]] and/or [[?MULTIHASH]] which have already replaced the normative references to [[MULTIBASE]] and [[MULTIHASH]], I expect the informative references will be restored).

msporny commented 11 months ago

@selfissued wrote:

Replacing the normative reference with an informative reference effectively leaves proofValue undefined. That's not OK, since it's a required field. This PR needs to be updated to normatively define the contents of the proofValue field.

Fixed in dd44d6b1513b96ae72c96ac245f65507290a301e.

msporny commented 11 months ago

"Based on the @type and cryptoSuite used by the proof, the specification defining the crypto suite can define the semantics of proofValue, the JSON value MUST be a string."

If we use a "string" datatype, it creates the following two issues:

  1. CBOR-LD compression of multibase values breaks because there would be no way to tell if the first character in the proofValue is a multibase header or just so happens to be a value that starts with something that looks like a multibase value.
  2. It just kicks the definition of the encoding of the proofValue to each cryptosuite, so we'd end up effectively repeating the same text here in every cryptosuite.

Then site base58btc normatively as the string encoding, but just copy the alphabet used in to the spec.

That alphabet exists in the spec today: https://w3c.github.io/vc-data-integrity/#example-an-integrity-protected-image-that-is-associated-with-an-object

... but I moved it into its own section to make it more clear about the normative definition based on what @selfissued was asking for. That should address @TallTed's concern about SHOULD vs. MUST (it's a MUST now because the spec provides the normative values).

I believe the latest changes should address each of your concerns, requesting a re-review from each of you.

iherman commented 11 months ago

The issue was discussed in a meeting on 2023-09-26

View the transcript #### 1.1. Remove normative dependency on Multibase and Multihash. (pr vc-data-integrity#196) _See github pull request [vc-data-integrity#196](https://github.com/w3c/vc-data-integrity/pull/196)._ **Manu Sporny:** If you could take another look at that PR, Mike -- and if Orie can look at that PR then that's the last thing to be addressed before we go forward with CR. … I guess we can talk about that more tomorrow, this is just a request for a re-review. **Kristina Yasuda:** Cool, ok. Put that on the agenda for tomorrow, Orie can join tomorrow. The controller document conversations in DI and vc-jose-cose -- if you could take a look at my last comment and respond that would help. **Manu Sporny:** Kristina, if you made that comment an hour or two ago I probably thumbs up'd it. **Kristina Yasuda:** After that. **Manu Sporny:** I'll look.
iherman commented 11 months ago

The issue was discussed in a meeting on 2023-09-27

View the transcript #### 2.2. Remove normative dependency on Multibase and Multihash. (pr vc-data-integrity#196) _See github pull request [vc-data-integrity#196](https://github.com/w3c/vc-data-integrity/pull/196)._ **Manu Sporny:** selfissued suggested we removed normative dependencies on any multiformats specifications, because they aren't standards. … I heard consensus during f2f where we agree to make those references non-normative. … This is what the PRs I raised are doing. … I think what selfissued is asking for is the complete removal to any reference. Even if they are non-normative. … As for the other issues raised with other cryptosuites, they say what the normative values should be. **Michael Jones:** The core of the discussion wasn't about normative vs non-normative references. It was about required features needing to be normatively defined. … They have to be defined before going to CR. > *Manu Sporny:* [https://pr-preview.s3.amazonaws.com/w3c/vc-data-integrity/pull/196.html#dfn-proofvalue](https://pr-preview.s3.amazonaws.com/w3c/vc-data-integrity/pull/196.html#dfn-proofvalue). > *Manu Sporny:* ^ normatively defined right there. **Michael Jones:** I'm still seeing that the value is defined, which cannot be defined since multiformats isn't a standard. … I think we should be able to normatively defined the contents of every spec without referring to the multiformats spec. … Adding a note stating that it's compatible is ok. **Kristina Yasuda:** To clarify, you are asking the DI document to redefine multibase/multicodec instead of referring to the internet draft? **Michael Jones:** Yes, with the subset that actually matters. **Manu Sporny:** That's exactly what it does. … `<`manu reads the pr preview`>`. > *Michael Jones:* "The contents of the value MUST be a [MULTIBASE]-encoded binary value with a header and encoding as described in 2.4 Multibase." retains normative use. **Kristina Yasuda:** Why do you need a reference to multibase? **Manu Sporny:** Because it's useful for implementers. Explains the background. … Useful as pointing to any non-normative document. > *Manu Sporny:* "[MULTIBASE]-encoded". **Manu Sporny:** I'm confused what's being objected to. **Michael Jones:** quoted the sentence that's objected to. **Manu Sporny:** I'll make the changes to remove the multibase bit. … The other PRs point to the section that will be fixed. … That's how they refer to it normatively. > *Michael Jones:* See [https://github.com/w3c/vc-di-eddsa/pull/63/files](https://github.com/w3c/vc-di-eddsa/pull/63/files). **Manu Sporny:** I'll make sure that all similarly worded phrases refer to the normative spec (i.e. DI). > *Kristina Yasuda:* same for [https://github.com/w3c/vc-di-eddsa/issues/62](https://github.com/w3c/vc-di-eddsa/issues/62) and [https://github.com/w3c/vc-di-bbs/issues/92.](https://github.com/w3c/vc-di-bbs/issues/92.). **Michael Jones:** That also has a sentence. **Manu Sporny:** I can remove that as well. … we have a path forward. **Sebastian Crane:** Pointing out the multibase isn't a protected trademark. … We should be specific and define everything that we need. … We can still call it multibase. > *Orie Steele:* [https://twitter.com/multibase_co](https://twitter.com/multibase_co) is a YC company, doing web3 analytics.... which i get ads from constantly. **Kristina Yasuda:** might be confusing, but that's fine. … got 3 minutes, any issue/PR? ---
iherman commented 11 months ago

With the current setup, §2.3.1.2 is a normative section that does not have normative references. I do not think that is o.k. I believe that this section must be flagged as informative or marked as "at risk". The latter may force us to remove the section if Multibase does not receive a normative stamp, which may harm already deployed software. I would therefore avoid that, and suggest marking the section as informative for now, making it clear (in a note) that if the IETF stamp is there, then this become normative (provided we have enough test results). Because publicKeyJwk is also a (normative) possibility to express the verification method, we do not really have a normative problem expressing that.

Having read the meeting minutes of the last call (see also https://github.com/w3c/vc-data-integrity/pull/196#issuecomment-1738307954) I realize I misread the text. The intention of the spec is to normatively define the portion of MULTIBASE that is used by us. That is fine. However, the normative definition should then not refer to the MULTIBASE document; the text should stand by itself. It may (should) refer to it informatively to show the background for reasons of compatibility, and it is also o.k. to refer to do it in the initial note. At the moment, it is not clear how the 'z' and 'u' characters should be used, for example (without going back to the MULTIBASE document).

My proposal is to

(The same comments apply to Multihash, b.t.w.)

OR13 commented 11 months ago

@iherman are you saying this URL needs to change in the v2 context ^

msporny commented 11 months ago

@selfissued @OR13 I have made the specific change requested on the call yesterday in this commit: https://github.com/w3c/vc-data-integrity/pull/196/commits/7bb2e688e4491a06f316e089877bee1799f84df5

I have also gone through and removed any other reference to [[?MULTIBASE]] or [[?MULTICODEC]] in normative statements as well to address your concerns in case they exist anywhere else in the specification:

b9d5d96dfb27d86b43063f56c49cdd181d3b212f

I also took @selfissued's advice and modified the issue marker to note that compatibility between what we're defining in the specification, the Multiformats community, and the I-Ds at IETF (which might go standards track in a future Multiformats WG) are a desired outcome.

@iherman I have also attempted to address your concerns in b9d5d96dfb27d86b43063f56c49cdd181d3b212f:

put there a complete, comprehensive definition for the features we use

I think this is done as of the latest commit?

give background information in the initial note, with a non-normative reference to Multibase

I think this is done as of the latest commit.

the feature should then NOT be at risk! The exact text may change, but the feature, as used in DI is not at risk at all...

Yes, agree.

the text in Datatype specification must also be changed: the datatype is defined via §2.3.1.2 and not via the Multibase spec (but that can also change later)

I have updated that text in b9d5d96dfb27d86b43063f56c49cdd181d3b212f

I think this resolves all issues raised on the call yesterday. Requesting re-review from @OR13, @selfissued, and @iherman .

iherman commented 11 months ago

@iherman are you saying this URL needs to change in the v2 context ^

I do not understand the question, @OR13.

iherman commented 11 months ago

@msporny,

put there a complete, comprehensive definition for the features we use

I think this is done as of the latest commit?

I tried to look at this, and I do not see any change. What I see (I downloaded your branch to my local machine) is:

Screenshot 2023-09-28 at 16 15 06

I try to understand the table without looking at the IETF document, and I do not understand. What does 'u' and 'z' mean, how they are used, how does the multibase value reall looks like? It is not said.

give background information in the initial note, with a non-normative reference to Multibase

I think this is done as of the latest commit.

You mean in the introductory issue box? +1 to that.

the feature should then NOT be at risk! The exact text may change, but the feature, as used in DI is not at risk at all...

Yes, agree.

But the issue box still says "Feature at risk". This should be removed (also in the multihash section)

the text in Datatype specification must also be changed: the datatype is defined via §2.3.1.2 and not via the Multibase spec (but that can also change later)

I have updated that text in b9d5d96

Yes, that looks good to me.

I think this resolves all issues raised on the call yesterday. Requesting re-review from @OR13, @selfissued, and @iherman .

OR13 commented 11 months ago

@iherman are you saying this URL needs to change in the v2 context ^

I do not understand the question, @OR13.

@iherman proofValue is normatively defined via the JSON-LD context to be:

"@type": "https://w3id.org/security#multibase"

W3C is not defining multibase as an RDF type, W3ID.org (a separate legal entity with some overlap in governance, but not equivalent to this working group or W3C) is defining multibase as an RDF type... @iherman I am asking you if this is a problem or not based on the other changes being made in this PR, from the perspective of W3C.

iherman commented 11 months ago

@iherman are you saying this URL needs to change in the v2 context ^

I do not understand the question, @OR13.

@iherman proofValue is normatively defined via the JSON-LD context to be:

"@type": "https://w3id.org/security#multibase"

W3C is not defining multibase as an RDF type, W3ID.org (a separate legal entity with some overlap in governance, but not equivalent to this working group or W3C) is defining multibase as an RDF type... @iherman I am asking you if this is a problem or not based on the other changes being made in this PR, from the perspective of W3C.

The definition of the multibase RDF Type is defined in the DI spec of W3C. It is an entry in the security vocabulary and, so far, the WG agreement is that this vocabulary's URI is https://w3id.org/security. There is nothing special about multibase in this respect. Put it another way, there is no difference between that datatype in the vocabulary and, say, the proof property. This PR has no effect on these whatsoever.

iherman commented 11 months ago

the definition of the multibase RDF Type is defined in the DI spec of W3C.

@OR13 , I forgot to make a reference to where this happens: look at §2.9.3 in the branch of this PR.

msporny commented 11 months ago

@iherman wrote:

I try to understand the table without looking at the IETF document, and I do not understand. What does 'u' and 'z' mean, how they are used, how does the multibase value reall looks like? It is not said.

This has been fixed in 2651487fc6fd4d29fa7d805d43f1b873ecadc42a and 02121e37066827d60bcb902c1dffa115484d4383.

You mean in the introductory issue box? +1 to that.

Yes.

But the issue box still says "Feature at risk". This should be removed (also in the multihash section)

Fixed in 55638820b0aace3b8c8fc4ef29ea4339c886c47f.

Does that address all of your concerns, @iherman ?

msporny commented 11 months ago

The latest changes answered all my concerns on this PR. To avoid the hassle of yet another PR, I would prefer to update the specification text for multihash in this PR, too, but I leave this to the editors.

That was done in https://github.com/w3c/vc-data-integrity/commit/02121e37066827d60bcb902c1dffa115484d4383

iherman commented 11 months ago

The issue was discussed in a meeting on 2023-10-04

View the transcript #### 1.3. Remove normative dependency on Multibase and Multihash. (pr vc-data-integrity#196) _See github pull request [vc-data-integrity#196](https://github.com/w3c/vc-data-integrity/pull/196)._ **Manu Sporny:** this is a base PR for many other things. … selfissued has made many requests. … and I want to get through them on the call. > *Manu Sporny:* See [https://github.com/w3c/vc-data-integrity/pull/196#pullrequestreview-1657894864](https://github.com/w3c/vc-data-integrity/pull/196#pullrequestreview-1657894864). **Manu Sporny:** 'cause this is dragging out. … selfissued, you are asking for more normative definitions where there are just examples. … so the normative references you want are in other specs. > *Manu Sporny:* [https://pr-preview.s3.amazonaws.com/w3c/vc-data-integrity/pull/196.html#multibase-0](https://pr-preview.s3.amazonaws.com/w3c/vc-data-integrity/pull/196.html#multibase-0). **Manu Sporny:** this preview ^ shows these definitions have been in there for awhile. … it's normatively defined in the specification. … the other thing is asking about the header format and how they're used. … they're headers, they're doing what they've been defined to do. … these are examples, so they are not normative. > *Manu Sporny:* [https://pr-preview.s3.amazonaws.com/w3c/vc-data-integrity/pull/196.html#resource-integrity](https://pr-preview.s3.amazonaws.com/w3c/vc-data-integrity/pull/196.html#resource-integrity). **Manu Sporny:** same thing is true for the others also. … this stuff is defined in there. … I can go and respond to each one, but they are all already normatively defined. **Michael Jones:** I am reading the specification. … the text that you're talking about does not normatively define how to translate the alphabet to string or binary. … in all cases where we're using data structures, we need to define in sufficient detail, so implementers can build implementations. … I'm still doing a review, but there's similar incomplete definitions in EDSA as well. … counting on tribal knowledge isn't acceptable when we're normatively referencing something. **Manu Sporny:** base encoding something is something no ones had a problem with since forever. … no other spec has to define this. **Michael Jones:** no. you must define it in the spec. > *Orie Steele:* `@manu`, you can paste in this example: [https://github.com/cryptocoinjs/base-x/blob/master/src/index.js.](https://github.com/cryptocoinjs/base-x/blob/master/src/index.js.) lol. **Manu Sporny:** then if I define base encoding, will you let us merge this? **Michael Jones:** yes. **Manu Sporny:** so you want all these things redefined directly in this spec? **Michael Jones:** yes. … there's a term, multibase prefix...and that's not defined. … when I review stuff, I like to pretend I'm an implementer. … and imagine whether I can build code with that of others. … and if I can't, then I raise issues. > *Manu Sporny:* Here is the normative definition: [https://pr-preview.s3.amazonaws.com/w3c/vc-di-eddsa/pull/63.html#multikey](https://pr-preview.s3.amazonaws.com/w3c/vc-di-eddsa/pull/63.html#multikey). > *Manu Sporny:* Here is the other normative definition: [https://pr-preview.s3.amazonaws.com/w3c/vc-di-ecdsa/pull/42.html#multikey](https://pr-preview.s3.amazonaws.com/w3c/vc-di-ecdsa/pull/42.html#multikey). **Manu Sporny:** I have pointed directly to the normatively to the definitions. … and they tell you how to build the format front to back. … I can make these changes, selfissued. … but every time I make a change, you request new ones. **Michael Jones:** I'm sorry that's the way it feels. … but the intent is to have each of these specs completely define everything implementers need to do to implement everything. > *Dave Longley:* would referencing RFC 4648 and say that the bases used are `58`, `64`, and the alphabets are `base58-btc` and `base64-no-pad`? > *Dave Longley:* would that help? > *Dave Longley:* [https://datatracker.ietf.org/doc/html/rfc4648](https://datatracker.ietf.org/doc/html/rfc4648). **Manu Sporny:** can you read what dlongley just shared. **Michael Jones:** maybe. I'd have to look. > *Orie Steele:* ref'ing that RFC seems like a good move. **Manu Sporny:** I can address all the comments. **Kristina Yasuda:** sorry we can't pick a date. … we'll continue to believe there's good intent here. … but please address these concerns selfissued. **Sebastian Crane:** it seems selfissued's main concern is anything multibase/multikey. … it seems our charter includes defining these things. > *Orie Steele:* +1 to potentially publishing multibase as a w3c document. **Sebastian Crane:** so, if selfissued, you take issue with these not being defined at the IETF, then a probably solution is to take these multibase/multikey stuff through the W3C process within this group. **Kristina Yasuda:** we can talk about multibase. … but I do not think it is realistic within the lifetime of the WG. … in the meantime if selfissued and manu can work through the PR that would help. … next week is IIW, but we will cancel the special topic call. … but keep the main call. … I think the special topic call will be about accessibility. … so seabass if you could send stuff out about it, that'd be great. **Sebastian Crane:** I will be absent for part of the coming weeks. **Ivan Herman:** next week, WG is at a EU unfriendly time. … and I want to be part of the timing discussions. … so you're welcome to meet and provide a time...but I may have to disapprove it after I see it. **Kristina Yasuda:** yeah, lets discuss offline. … thx all. … thx for scribing bigbluehat. ---
msporny commented 10 months ago

@selfissued re-ping to re-review... ideally, you approve before we merge this. It's all green on reviews so far.

iherman commented 10 months ago

The issue was discussed in a meeting on 2023-10-11

View the transcript #### 1.1. Remove normative dependency on Multibase and Multihash. (pr vc-data-integrity#196) _See github pull request [vc-data-integrity#196](https://github.com/w3c/vc-data-integrity/pull/196)._ **Manu Sporny:** Mike Jones had requested that we normatively define a number of things in multibase/multihash. The changes have been to define how base encoding works and provide algorithms for base encoding and decoding and specifically provide the alphabets for the bases used. … Those are the updates and changes and that has gone into PR #196. … I believe I have addressed every single one of Mike Jones' requests and questions and AFAICT everything is normatively defined. The hope is to get his review in and get this PR in before Nov 7. **Brent Zundel:** Merge it. … Mike has approved.
iherman commented 10 months ago

The issue was discussed in a meeting on 2023-10-11

View the transcript #### 1.2. Remove references to Multiformats specifications. (pr vc-di-eddsa#63) _See github pull request [vc-di-eddsa#63](https://github.com/w3c/vc-di-eddsa/pull/63)._ **Manu Sporny:** That is the basis for the other two PRs. _See github pull request [vc-data-integrity#196](https://github.com/w3c/vc-data-integrity/pull/196)._ **Manu Sporny:** PR #63 builds on DI PR #196. My presumption there is if Mike is good with 196, he's good with the others that refer to it. … He has +1'd that as well.
msporny commented 10 months ago

Normative, multiple reviews, changes requested and made, no objections, merging.