Spec writable protocol handlers

[RangerMauve]

As per @lidel's request here's an issue to track the discussion we're having on standardizing writability in IPFS protocol handlers.

The tl;dr, we've got ipfs:// and ipns:// protocol handlers in browsers like Agregore/Brave/Capyloon and most of them focus on what it means to GET some data from the protocol.

In Agregore's case, we've also had the ability to PUT or POST data to IPFS.

Initially this was done via the POST verb which would take some data, upload it to IPFS, then return the uploaded IPFS URL in the response body with a 200 status code.

Earlier this year, during the Agregore Mobile devgrant, we forked Kubo's writable gateway in order to get IPFS protocol handlers on Android with a Chromium fork. Part of this effort involved writing a spec based on Kubo's writable APIs.

The main thing that changed was that Kubo was using the 201 status code and placing the resulting CID in the Location header instead of into the response body. Initially I was aprehensive and wanted to use what we already had, but @makeworld-the-better-one convinced me using solid reasoning which I no longer remember that we should switch over.

Most recently, @fabricedesre was talking about wanting to move back to the 200+body approach for showing result URLs. It seems like this might be the apoproach he'd like to take for Capyloon which is now embedding an Iroh node which could lead to local authoring of data.

Personally, I'd like to keep the 201+location method if possible since I have code which relies on that already, but it's not a hard blocker and I'd be willing to update stuff if it means we can get writability into other browsers sooner.

This also relates to discussions we're having for ipld protocol handlers where we need to specify what an IPLD POST/PATCH should yield.

Ideally we will book a time during Labweek 2022 in Lisbon to sit down and talk about this stuff to figure out what approach we align on.

[makew0rld]

• 200 when an object is created and returned
• 201 when an object is created but only its reference is returned (such as an ID or a link)

https://stackoverflow.com/a/44396004

This is the motivation behind using 201 instead of 200 as I recall, it's more semantically accurate. It's also possible the kubo writeable API was already using 201 in some places, so I continued with that in the spec? Not completely sure right now.

It seems to me that 201 is more correct, but at the end of the day using 200 shouldn't cause any problems.

[lidel]

Kubo's writable gateway

Just noting this is an ancient experiment, which is a good starting point, but should by no means limits the design of the final API. It was created in the very beginning of IPFS, when we only had UnixFS and gateways had no Block/CARs.

I suggest we start with a blank canvas. Below is my quick brain dump – food for thought before Lisbon.

Key design requirements

Whatever we design, must be possible to map well between HTTP gateway and native protocol handlers.

When we design a modern version, I believe we need to cover a specific minimal set of things:

• Ingesting data, chunking if needed, and producing a CID for it
  • a File
  • unixfs / raw by default
  • smart handling of application/json and application/cbor
  • smart handling of application/vnd.ipld.dag-json and application/vnd.ipld.cbor
  • arbitrary directory tree (as a TAR stream) application/x-tar
• Ingesting already content-addressed data, verifying, but accepting precalculated CID as-is
  • A Block application/vnd.ipld.raw
  • A bag of Blocks (CAR stream) application/vnd.ipld.car
• Patching existing IPLD data
  • sending https://ipld.io/specs/patch against existing CID, getting CID for patched version back (content type tbd, maybe application/vnd.ipld.patch)

We don't need to spec or implement everything at the same time, but we need to have these use cases in mind, to be sure we are not sabotaging future with short-term decisions.

[ikreymer]

If the primary application is access to a local and/or embedded ipfs mode by a webpage in a browser, then I think the design needs to include the the security / permissions model for a website pinning data to a local node. Otherwise, no matter what the POST/PUT API is, a browser vendor can't be expected to just allow any website to connect to a localhost port and POST/PUT unlimited data.

Some ideas include extending navigator.permissions.query and then perhaps another api to get an auth token that is passed to all APIs. A next step would also be doing something with the quota system. All of this is a bit more tricky, but probably worth figuring out to be able to support true p2p ipfs access in browsers. (Maybe this should be its own issue, though :)

[RangerMauve]

Still into this. Any clue how I can book time at PL Summit? Wednesday I'm doing some IPLD roadmapping, but I'm free outside of that.

[RangerMauve]

Looks like some time during IPFS Camp would be good. Maybe a chat during breakfast before one of the days.

[RangerMauve]

Might do Friday?

[RangerMauve]

Notes from meetup at Lisbon:

• We should have content-type as part of the data when uploading @darobin Should be strict with top level content type and how it's processed
• Posgt to ipfs://localhost to upload a binary or a formdata
  • Might want to bikeshed localhost
• PUT to ipfs://CID/path to add a file or upload a folder
• Would be useful to think about authorization for the gateway use case @gozala has thoughts from web3.storage
• Would be nice to spec CAR file upload soon after this initial thing
• Using redirects in the response can lead to a bunch of issues, should return the URL itself
• 201 is useful in that it's semantically correct, and in a form submission it doesn't navigate away.
• Should have the PUT over data be a separate spec, then stuff like upload APIs on gateways won't need to bother with this

[RangerMauve]

One thing that would be ueful for IPFS is to import public keys from WebCrypto

User agent (browser) can decide to provide users with a UI to manage key access

Capolyoon is down to experiment with soem initial IPFS//IPNS upload APIs

[RangerMauve]

Will work on an initial spec PR for protocol handlers and go from there.

Might be useful to have stuff about where to save things for the gateway use case

[RangerMauve]

@darobin would you be down to have a call some time soon to start working on the spec together? Maybe we could have an agenda item on the browsers and standards call? cc @autonome would that work? Would also be good to have Fabrice in there

[darobin]

Sure thing. I've been going through the existing specs and also looking at what Kubo does. I think having the discussion with the whole gang might be the best option.