CBOR tag conflict :(

[mikeal]

Hello from the IPLD project ;)

For a few years we’ve been using CBOR tag 42 in dag-cbor. It’s actually the only tag we use, and we use it to note when something is a link to another piece of data. At the time we started using it there was nothing registered but we noticed that the IANA registry recently assigned it to this spec.

It would be very hard for us to migrate given the amount of data already created and persisted in the network.

[cabo]

Well, the point of a registry is that you can register code points so they are not taken from you.

Can you tell us more about the hardship you will be experiencing? If you make a very good point, maybe we can still fix this.

[lemikev]

Hi Carsten

Changing these tags on our side is not too difficult. If we decide to change them, the decision need to be done quickly and applied to the IANA site and the draft.

Ivaylo, I suggest to postpone the revision of the YANG to CBOR draft until a decision is taken.

Regards, Michel

From: cabo notifications@github.com Sent: 1 août 2019 15:23 To: core-wg/yang-cbor yang-cbor@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [core-wg/yang-cbor] CBOR tag conflict :( (#13)

Well, the point of a registry is that you can register code points so they are not taken from you.

Can you tell us more about the hardship you will be experiencing? If you make a very good point, maybe we can still fix this.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcore-wg%2Fyang-cbor%2Fissues%2F13%3Femail_source%3Dnotifications%26email_token%3DADZVLN766ALB27DP6RRXUDDQCMZZFA5CNFSM4IITHQQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3LUCVA%23issuecomment-517423444&data=02%7C01%7CMichel.Veillette%40trilliant.com%7Ceda0562657bb48daadc408d716b5a993%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C0%7C637002841808318493&sdata=eqckM4vSwM9Z0RZvhacnIqwuBpg5fbt3u2A39nScEYo%3D&reserved=0, or mute the threadhttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADZVLN6YEQ72V3MP5U2GSATQCMZZFANCNFSM4IITHQQA&data=02%7C01%7CMichel.Veillette%40trilliant.com%7Ceda0562657bb48daadc408d716b5a993%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C0%7C637002841808318493&sdata=%2Fy5QOJeWa1P9oXOmkVp5Adk6anE%2B1684s3KI9%2FAy%2FSM%3D&reserved=0.

[vmx]

Here's a bit more background on this. We use Tag 42 to identify CIDs. Those CIDs are constructed out of several other specifcations. The plan is to get the CID (as well as the other specs) through the IETF standards process. Some early drafts were already submitted (multihash and multibase).

[cabo]

Thank you. Two more questions:

(1) is there a specification (specifications are required for 1+1-byte tags)? (2) can you say more about the installed base that would be invalidated by using a different tag number for CIDs?

[vmx]

(1) is there a specification (specifications are required for 1+1-byte tags)?

I just read the RFCs about what "Specification Required" means. If I understood it correctly it doesn't need to be an IETF one. I would hope that this one: https://github.com/multiformats/cid/ is good enough to count as such a spec.

[cabo]

(1) is there a specification (specifications are required for 1+1-byte tags)?

I just read the RFCs about what "Specification Required" means. If I understood it correctly it doesn't need to be an IETF one.

That is correct.

I would hope that this one: https://github.com/multiformats/cid/ is good enough to count as such a spec.

I think so (well, maybe you could improve the references a bit), except that it doesn't say anything about how you use the CBOR tag.

[Stebalien]

That spec is here: https://github.com/ipld/specs/blob/master/block-layer/codecs/DAG-CBOR.md. The CID spec just defines the structure of the type we're tagging with 42.

[vmx]

(2) can you say more about the installed base that would be invalidated by using a different tag number for CIDs?

The biggest project which is using IPLD is certainly IPFS. A CID is used to identify the data. There's is thousands of nodes running (I should be able to get a better estimate if that's needed). Not all data is encoded as CBOR, depending on how you use it, you can choose which codec you want to use. One of them is CBOR. Though if you choose to encode data as CBOR, then Tag 42 will be used if you store a link to another object (hence a CID).

There are more projects using IPLD, I currently try to get numbers about their deployments.

[ianopolous]

We build an app, Peergos, on top of IPFS which has many users and everything we do is in cbor using the 42 tag. Users sign the merkle root of cbor trees so we can't even migrate users data for them because we don't have their keys.

Our implementation of the cbor encoding using the tag is: https://github.com/Peergos/Peergos/blob/master/src/peergos/shared/cbor/CborObject.java#L245

[ivajloip]

@lemikev no problem on my side to wait a bit. I will check the merge that we discussed today, but I am not going to publish a new draft revision right away.

[mikeal]

Sounds like changing the tag on your side is doable. Is there anything else you need from us in order to facilitate this change?

We’re working on spec improvements on our side as well so that once the change lands we can request the tag in the IANA registry and avoid this problem surfacing again in the future.

[cabo]

I have asked to discuss this at the CBOR WG virtual interim at 1500Z today. ⁨https://mailarchive.ietf.org/arch/msg/cbor/m3sTRvBgdRGnJpxfv1a8X1uRiGM⁩

My summary of the situation is at: ⁨https://mailarchive.ietf.org/arch/msg/yot/Q0xVcC40K1mWIT4tKcqt2w7PRV0⁩

[abierman]

Now you know why I advocated for YANG Hash at the start of this work. You have not even published an RFC. You have not even allocated 50 SIDs. And already there are conflicts because humans do not follow procedures.

[vmx]

@cabo Would it help if someone from the IPLD team joins the call (are outsiders allowed/welcomed)? I could join if it makes sense.

[cabo]

@vmx I'm sorry, I didn't see the comment until today.

We briefly discussed this at the Interim and the sentiment was favorable for moving yang-cbor out of the way. There also was some mild agreement that you guys should be slapped somewhere for not getting this out earlier...

The next step, of course, is doing a proper registration of 42. Note that the template is:

o Data item (so, what can be in the tag? A byte string, I suppose.) o Semantics (short form) (a couple of words so unrelated people can quickly find out what that is) And, given that this is a "specification required" registration, a pointer to a stable, long-term URI with a specification that people can use implementing the required functionality (and that probably contains the information from the template); this is probably a little piece of work that still needs to be done.

Compare https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml for the four columns that ultimately will be registered. Note that we have had a relaxed view of the stable URL in the past; there are voices arguing for getting a bit more formal here. But for now a github org-based repo should be good enough.

[cabo]

https://github.com/ipld/specs/blob/master/block-layer/codecs/DAG-CBOR.md is not really a good thing to reference from the tags registry. A simple document explaining what the tag is (that can then reference DAG-CBOR.md as well as the multibase format (?)) like the other ones in the CBOR tags registry already, would do. (And, yes, please do get rid of the arbitrary map key restrictions.)

I'm not sure what you use canonical DAG-CBOR for. Note that we are simplifying the rules for deterministic encoding a bit in the successor for RFC 7049, "7049bis". (But the old canonical format will stay as the less desirable alternative, of course.)

[cabo]

@abierman and nobody has made a mistake with SIDs yet, only with CBOR tag numbers, and these have been out for almost six years now :-) Yes, getting that right will be a fun ride. I'm a bit more optimistic than you, though.

[abierman]

Sorry I was confused. SIDs are already heavy-weight on the client because each server can define its own experimental range. The client already has to maintain a SID registry per session for 1 range of SIDs. The client may be able to trust that the SID file for one server is correct. It may not trust that servers are using the correct global SID such that the client can hard-wire common code with those SIDs.

[vmx]

We briefly discussed this at the Interim and the sentiment was favorable for moving yang-cbor out of the way. There also was some mild agreement that you guys should be slapped somewhere for not getting this out earlier...

Wow, thank you sooo much! I accept the slap in the name of the IPLD Project and fully agree that we should have done this earlier.

I will make sure that we have a document that fits those requirements soon.

[cabo]

IANA was kind enough to execute the move away from 42 yesterday: https://www.iana.org/assignments/cbor-tags

So this change needs to be reflected in the yang-cbor draft (which needs to change to language like "IANA has allocated" anyway).

And the 42 shouldn't stay "unassigned" for too long...

[vmx]

@cabo Thanks for letting me know.

I've been working on a spec yesterday, do you think this spec is good enough to be accepted for getting tag 42 assigned: https://github.com/ipld/cid-cbor

What would be the next steps?

[cabo]

On Aug 20, 2019, at 12:12, Volker Mische notifications@github.com wrote:

@cabo Thanks for letting me know.

I’ve been working on a spec yesterday, do you think this spec is good enough to be accepted for getting tag 42 assigned: https://github.com/ipld/cid-cbor

That is quite close already. I’d probably be a bit more explicit in (1) the heading, (2) the first sentence, and (3) the actual registry entry, so people who start at the registry immediately know where they are:

Content identifiers (CIDs) in CBOR ➔ IPLD/IPFS Content identifiers (CIDs) in CBOR

This document registers a tag for serializing content identifiers (CIDs) in Concise Binary Object Representation (CBOR). ➔ This document registers a tag for serializing IPLD/IPFS content identifiers (CIDs) in Concise Binary Object Representation (CBOR).

Semantics: Content identifier ➔ Semantics: IPLD/IPFS Content identifier

(I don’t know whether “IPLD/IPFS Content identifier” is the right long-form term, but you see what I’m trying to do here.)

I’d probably also add a date, even though that is also implied by the github commit.

(And you’ll see I have had too much contact in my life with copy editors when I suggest s/like/such as/.)

This spec relies heavily on references which I hope also are stable. https://github.com/multiformats/cid does not really give a fully unambiguous rule (e.g., it says a CIDv0 is always base58btc, but then it also defines a binary form), maybe one more critical reading would help. BTW, I don’t understand the “2^31” in https://github.com/multiformats/unsigned-varint — this should be 2^63? (And, nitpicking, “larger than” is >, while you want ≥.)

What would be the next steps?

When you are happy internally that the spec is stable, and so is the URI, I’ll relay this to IANA (you could do that, but then they’ll play it by me, so this is a bit of a shortcut).

Grüße, Carsten

[vmx]

Thank you so much @cabo for the feedback. I've updated the repo, we call it "IPLD content identifiers" now. Also your comments on the other specs were incorporated.

The CID spec is also pretty stable. That's also the one we want to get properly drafted within the IETF in the future.

So please take https://github.com/ipld/cid-cbor/ as the spec to register tag 42.

[cabo]

And, thanks to IANA, now the new meaning of tag 42 is registered as well: https://www.iana.org/assignments/cbor-tags

So this closes the issue for the IPLD/IPFS folks. I'm keeping this issue open until the yang-cbor document (which, after all the issue is on) is edited to reflect the move from 42 to 47 as well; this will have to wait until France returns from vacation.

[vmx]

@cabo Thank you so much again. For all the help and getting it done so quickly!

[cabo]

Forgot to close this -- this has been resolved with a nice outcome.