search

ipfs / in-web-browsers

Tracking the endeavor towards getting web browsers to natively support IPFS and content-addressing
https://docs.ipfs.tech/how-to/address-ipfs-on-web/
MIT License
345 stars 29 forks source link

IPLD support on Gateways #182

Open lidel opened 3 years ago

lidel commented 3 years ago

Current state

Gateway provided by go-ipfs 0.8.0 supports only dag-pb (unixfsv1) and raw (raw block used for leaves) codecs.

Requesting any other IPLD type over a gateway fails.

Where we want to be

It should be possible to download everything over Gateway:

Low-hanging fruit: traversable JSON/CBOR documents

Some ideas how to maximize the utility of gateways (those are just prompt for discussion, details TBD):

Ongoing work

Open questions

  1. Is CAR something we want to introduce, or do we want to wait for "CARv2" like anorth/go-dar
  2. Should it be possible to include non-unixfs nodes inside of a unixfs directory? (this impacts MFS and ipfs-webui)
  3. What should be the default response type for dag-cbor? User will be able to choose, but what happens when there is no user preference? (Original binary or should Gateway return JSON as its more user friendly and makes onboarding easier?)
    • I was initially locked on keeping original format at all cost, but I now see how returning JSON for dag-cbor by default make it work out-of-the-box in browser after copying and pasting the CID, which makes our stack "feel" approachable and easy to understand. This is huge for onboarding new users (devs).
  4. Should we support graphql-like queries against dag-cbor, so only specific fields are returned (think /ipfs/{dag-cbor-cid}?keys=image,name)? (This is separate from traversing CID tags in CBOR)
    • This could be a hidden killer feature when it comes to building web apps against the gateways. Any reason to not do this? Does this clash conceptually with planned support for selectors?
  5. Do we need to bikeshed how the format parameter should look like on Gateway?
    • This will be something that can be added to every path to override the default representation, used in both https://github.com/ipfs/go-ipfs/pull/8037 AND https://github.com/ipfs/go-ipfs/issues/7552
    • We already support --enc=json on CLI, ?enc=json is short and easy to add by hand to URL - trivial important for dx/ux
    • If we plan to support responses other than dag-json and dag-cbor, we make it accept name from multicodec/table.csv
      • If we go with multicodec table, ?codec= or ?format= may be better than ?enc= (are they?)
  6. :pointright: (i am sure there is more, please comment below)_

cc @warpfork @alanshaw @Stebalien @aschmahmann

anorth commented 3 years ago

Enthusiasm for CARv2 over DAR seems thin. cc @mikeal @rvagg

rvagg commented 3 years ago

Enthusiasm for CARv2 over DAR seems thin. cc @mikeal @rvagg

No, just enthusiasm over messing with CAR at this stage is a little thin, but none of that discussion should be considered a blocker, there's no gatekeepers on this. Maybe the experimental work on Lotus integration for either CARBS or DAR will help resolve this a little? If it turns out DAR unlocks value over there then maybe we double-down, make it a CARv2, write a spec and create an upgrade path where it matters. Either way though, there will have to be an upgrade path so integration in to our suite of libraries will be important and gateway export could use either (or CAR now and CARv2 later).

Re response types and the discussion in ipfs/go-ipfs#8037, it would be really unfortunate to reinforce the JSON format that go-ipfs has relied on today over the more formal DAG-JSON form. Now that it's easier to interact with go-ipld-prime in go-ipfs, that will hopefully be unlocked? Being able to pass a ?format=X for any node would be interesting, where X could be dag-json which could even apply to raw and unixfs blocks but also any other format that can go through go-ipfs. X could be car too.

lidel commented 3 years ago
Gozala commented 3 years ago

@lidel from the description I do not really understand what the default content type is expected to be when navigating to e.g.

https://ipfs.io/ipfs/bafyreiahmfp4l6ii7h4ceumxdpaxwss5dldbevuvftjokcm6tx26gtgriy

Right now it's 404 with response body: ipfs cat /ipfs/bafyreiahmfp4l6ii7h4ceumxdpaxwss5dldbevuvftjokcm6tx26gtgriy: unknown node type which is probably the worst option.

I would argue that default should be along the lines of what is displayed by https://explore.ipld.io/#/explore/bafyreiahmfp4l6ii7h4ceumxdpaxwss5dldbevuvftjokcm6tx26gtgriy

As it would provide a much better experience in navigating non unixfs dags. Explorer view could be further improved to show file listing when links are dag-pb.

I think this still aligns with making gateway usable, as we Accept header / query parameter could be used to opt-in into data view as opposed to html view of things.

I think having such a view would also help us make a future case for encoding NFT as IPLD Dags that have IPLD links to all the assets as opposed to json file that happens to have ipfs:// style links to assets.

lidel commented 3 years ago

@Gozala Yes, it was unclear because we are trying to figure it out. I think you have a valid point: we already provide a basic GUI for directory listings, we should replace/improve it with something that works with non-dag-pb nodes. This sounds like a project on its own tho. Would you or someone from IPLD team be interested in writing a proposal for it? I think people close to IPLD should drive it, I'll be happy to help with go-ipfs/gateway side of things.

Anyway, we should not wait with CBOR support until we have UI like that.

Let's return informative error to retry the request with explicit content type, as suggested in https://github.com/ipfs/go-ipfs/pull/8037#issuecomment-815818230 This enables us to add GUI in the future, without breaking anything that people build on top of CBOR-as-JSON responses.

Gozala commented 3 years ago

Would you or someone from IPLD team be interested in writing a proposal for it? I think people close to IPLD should drive it, I'll be happy to help with go-ipfs/gateway side of things.

I would like that and I would have said yes, but I was already told too put all the proposals related work on the side, so I'm afraid I can't responsibly take this unless some resource relocation takes place.

jonnycrunch commented 3 years ago

shouldn't the logic happen in the http request header:

curl -H "Accept: */*" "https://ipfs.io/ipfs/bafyreidykglsfhoixmivffc5uwhcgshx4j465xwqntbmu43nb2dzqwfvae"

where cid is : base32 - cidv1 - dag-cbor - sha2-256-256-785197229dc8bb1152945da58e2348f7e279eeded06cc2ca736d0e879858b501

inspects the multicodec, which is 0x71 ( dagcbor) and return native format:

should return: 

*   Trying ::1...
* TCP_NODELAY set
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET /ipfs/bafyreidykglsfhoixmivffc5uwhcgshx4j465xwqntbmu43nb2dzqwfvae HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Clear-Site-Data: "cookies", "storage"
< Content-Type: application/cbor; charset=utf-8
< Location: http://bafyreidykglsfhoixmivffc5uwhcgshx4j465xwqntbmu43nb2dzqwfvae.ipfs.localhost:8081/
< X-Content-Type-Options: nosniff
< Date: Fri, 09 Apr 2021 15:34:12 GMT
< Content-Length: 94
<

* Connection #0 to host localhost left intact

Then the browser, if it understands application/cbor or ideally application/dag+cbor can consume it and download it etc.

Following on this, therefore the if the request header was different:

curl -H "Accept: application/json" "https://ipfs.io/ipfs/bafyreidykglsfhoixmivffc5uwhcgshx4j465xwqntbmu43nb2dzqwfvae"

even though the cid is dag-cbor, the request want application/json explicitly. Also doing this in the parameter ?enc=json seem redundant. Just need to register the dag+cbor as a MIME type. 

*   Trying ::1...
* TCP_NODELAY set
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET /ipfs/bafyreidykglsfhoixmivffc5uwhcgshx4j465xwqntbmu43nb2dzqwfvae HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.58.0
> Accept: application/json
>
< HTTP/1.1 200 OK
< Clear-Site-Data: "cookies", "storage"
< Content-Type: application/json; charset=utf-8
< Location: http://bafyreidykglsfhoixmivffc5uwhcgshx4j465xwqntbmu43nb2dzqwfvae.ipfs.localhost:8081/
< X-Content-Type-Options: nosniff
< Date: Fri, 09 Apr 2021 15:34:12 GMT
< Content-Length: 94
<
{"hello" : "world"}
* Connection #0 to host localhost left intact
Stebalien commented 3 years ago

Gateway provided by go-ipfs 0.8.0 supports only dag-pb (unixfsv1) and raw (raw block used for leaves) codecs. Requesting any other IPLD type over a gateway fails.

This is very misleading. IPFS supports unixfs (files) over the gateway, speaking of this in terms of IPLD codecs makes no sense (I can't, e.g., get a raw dag-pb node, etc.).

It should be possible to download everything over Gateway:

What about the read-only API?

Where we want to be

We do? Why?

I'd love a feature like this. I just want to make sure we're all on the same page about what "this" is, why we care about "this", etc.

  1. What problem are we trying to solve here? What use-cases are we trying to enable, why are existing features not sufficient, etc.
  2. What other solutions have we considered, etc.
lidel commented 3 years ago

Created https://github.com/ipfs/go-ipfs/issues/8234 with initial design proposal for /ipfs/{cid}?format=car|block|dag-cbor|dag-json|.. that aims to streamline all those threads under something that is intuitive to use and extend.

lidel commented 2 years ago
RangerMauve commented 2 years ago

Would it make sense to have a /ipld/ path on gateways for loading arbitrary IPLD data and for future support for POST /ipld/ similar to the writeable gateway spec for IPFS?

lidel commented 2 years ago

Good question! Whatever happens on gateways, the aim is to have it work with "native" schemes too. To reduce implementation overhead across ecosystem, we have the same abstraction on /ipfs/ and ipfs:// etc.

Theipld:// is not a thing atm, but my current thinking is that we will have IPLD support on /ipfs/ via ?format={ipld} or Accept header (wip: https://github.com/ipfs/go-ipfs/pull/8758)

In case of writable gateways, what we could implement is POST /ipfs/ that is sent with Content-Type: application/vnd.ipld.dag-cbor (or to /ipfs/?format=dag-cbor)

jonnycrunch commented 2 years ago

Would it make sense to have a /ipld/ path on gateways for loading arbitrary IPLD data and for future support for POST /ipld/ similar to the writeable gateway spec for IPFS?

Oh, now that would be interesting!

jonnycrunch commented 2 years ago

Theipld:// is not a thing atm, but my current thinking is that we will have IPLD support on /ipfs/ via ?format={ipld} or Accept header (wip: ipfs/go-ipfs#8758)

Let's make 'ipld:' a thing!

RangerMauve commented 2 years ago

I've floated ipld:// by @warpfork a bit and I also think it'd be useful to have coming from making apps in Agregore 😁

lidel commented 2 years ago

I am open to making it happen, but the main question we will hit with bigger vendors and developers will be: what is the difference between ipfs://, ipns:// and ipld:// ?

Right now, the story is clear:

What do we gain by introducing a second immutable namespace? Which behaviors would be different?

RangerMauve commented 2 years ago

I was thinking /ipld/ is for interacting with raw IPLD data, and /ipfs/ is for interacting with UNIXFS-like data.

E.g., doing a PUT to /ipfs/ assumes adding a file to a UNIXFS dir and the body is the content of a file, doing a PUT to /ipld/ requries it to be in some sort of IPLD-supported codec (maybe in the content-type header?).

Also potential for using the new IPLD patch spec for PATCH methods.

Some stuff that might be IPLD specific is stuff like Schemas, the different DAG traversal syntaxes and stuff like that.

At the moment it's a lot harder to do this fancy stuff with IPLD, and putting it into /ipfs/ where pretty much everything is file related doesn't feel as great.

Might be cool to brainstorm something more concrete if three's interest. 😁

RangerMauve commented 2 years ago

Started putting together a PR here: https://github.com/ipfs/specs/pull/293

Comments on current direction would be very much appreciated. Note that a lot of this is based on stuff from this initial exploration report: https://github.com/ipld/ipld/blob/e6cfab631d2bd24bf158d3a85e126514c98de5ce/notebook/exploration-reports/2022.03-ipld-url-scheme.md#future-work