Propose dag-pb-enc specification.

[kevincox]

This aims to add inline encryption to IPFS, particularly to the UNIX FS data model but all formats built ontop of dag-pb-enc are supported.

See requests and discussion on https://github.com/ipfs/notes/issues/270

[rvagg]

Oh, interesting. This could be tough to get over the line given the lack of interest in leaning too much more on dag-pb, but certainly worth discussing. I'm sorry to do this but would you mind closing this and opening it over at https://github.com/ipld/ipld. It's not well documented (yet) but we're retiring this repo and have moved the majority of the content there which is now available at ipld.io. You should see the codecs folder inside the specs folder where this will fit like the others.

[warpfork]

Procedurally: Yes, as @rvagg says, we've consolidated most content to the ipld/ipld repo.

Also procedurally, some expectation-setting: fair warning, the bar for merging anything like this directly into the documentation about codecs will be very high. There are several reasons for this:

• The pipeline for things to move from "rough proposal" to "serious enough to request a multicodec indicator code" to "serious enough to publish the specification on the main site" and then to "serious enough to get a first-class spot in the main documentation that actively directs users to the thing" is long. The latter stages are even longer.
• The bar will be much higher for anything regarding encryption.
  • Encryption is complicated. It's hard (both in time cost, and in skill required, and in care required) to review.
  • Encryption is high stakes. People (rightly) assign high value to encryption. That also means they assign huge value to any mistake in an encryption system, or, even any mistake in communicating what to expect from a cryptosystem. As a result, the cost associated with making a mistake with encryption (or recommending a system in which someone else might have made a mistake) is very, very high. Accordingly, we're going to be deeply cautious of what kind of claims and promises we'll host regarding any cryptosystem.
  • It is very difficult to recommend one encryption strategy. Often there are tradeoffs and often they are application specific. An exceptionally large amount of communication honing and review would be needed. We're not going to bless any one thing over another without great, great care.
  • Last and least: I'm (speaking only for myself on this) not clear on codecs being the right layer for solving encryption at all. I'd really like to see us exploring how encryption could work as an ADL, rather than as a codec :) That doesn't mean we can't also entertain encryption codec proposals, but I do warn you I'll bring this up and want to talk about whether the goals could be accomplished in that fashion before blessing any more encryption codecs.
• Note that we shifted stance on how to document things: we no longer usually give proposals the same footing as things that are exceptionally well known. Proposals usually go somewhere else in the navigational tree. Coming up are some suggestions for that.

Here are some milder steps that would be easier to take, and may help move in a forward direction:

• Option 1: If you want to give the notes a home, the notebook directory might be appropriate; this seems like content that could file as some sort of "exploration report". Making this content discoverable as an exploration report could be a good incremental step to make the work at least become discoverable.
• Option 2: If you want to develop this system on your own, but make sure eyes get to it: you can make a PR to the ipld/ipld//docs/synthesis/encryption/ page and add links to where you develop the spec or code.
  • :star: This is where the majority of other encryption systems we're aware of are being tracked right now. :star:
• Option 3: If you think this is a spec already, then the place to route it is towards the ipld/ipld//specs/codecs/ directory.
  • This should be treated as a big "if". The bar for this will be getting high, as warned above.
  • Note that all the other things in this folder have implementations before we've started accepting their specs here. We'll probably ask similar for any new contributions.
  • Note that by the time things start appearing here, they are also implicitly coming fairly frozen. If there's both a spec about it, and a multicodec indicator code assigned, then the expectation we want to cultivate for that is "the mapping from serial form to data model is now a fact that needs no additional versioning". So: don't commit to this too early. :)

I'd recommend starting with option 1 or 2. Shooting straight for option 3 is pretty extreme.

---

On the content: 2cents: I'd also probably be interested in seeing something based on CBOR wire format rather than based on Protobuf wire format. I suspect this could be written in either wire format without much actual semantic difference, and unifying more things on CBOR is something that generally makes us happier. I haven't reviewed this much deeper than that yet. A compare/contrast with other systems listed in https://ipld.io/docs/synthesis/encryption/ would also almost certainly help a great deal.

[kevincox]

Thanks for the feedback. The docs are very unclear so I'll just follow what you are suggesting. Sounds like I will update this doc to fit into option 1 and submit it there.

[kevincox]

I'd also probably be interested in seeing something based on CBOR wire format rather than based on Protobuf wire format. I suspect this could be written in either wire format without much actual semantic difference, and unifying more things on CBOR is something that generally makes us happier.

You are correct, it doesn't make any meaningful difference. I just updated the dag-pb format since it appears to be what the current unix-fs is based upon and unix-fs is what I am most interested in.

[kevincox]

Moved to https://github.com/ipld/ipld/pull/135