dag API should let a user ask for the daemon to fetch data matching a selector

[warpfork]

Feature description:

The ipfs dag API should let a user ask for the daemon to fetch data matching a selector.

• The data should become local to the daemon.
• We may need a way to wait for this.
• We don't necessarily need all that data streamed back from the same API invocation (!).
• We don't necessarily care what the transport behavior is that the daemon uses internally. (Maybe it's graphsync, and that's probably a good fit since it already consumes selectors -- but the end user doesn't necessarily care, just wants this to be fast, and work.)

Motivation:

Users might have large data structures that they want to work on, and want to get these locally before starting to work with them, for performance reasons; or, want to get them locally simply so they can confidentally go offline.

Ceramic has a good example story like this:

• Ceramic's "Streams" are a fairly long chain-like structure -- so fetching them iteratively can be somewhat slow (it will pipeline stall and experience network latency repeatedly as it's going down the chain).
• Being able to request large amounts of this chain from a peer at once would speed things up significantly.
• At the same time, Ceramic already has other code that talks to an IPFS API iteratively... but on a much lower latency connection than is involved in a network peer fetch. So this part is fine staying as it is.
  • (They could also use a CAR API endpoint... but don't need to. And sometimes may not actually need to review all of the data they fetch in one time, so the "fetch this" and "show me this" are sometimes different steps.)

Disambiguation:

This is very similar and related to some plans for having APIs that return CAR files (because we figure those APIs should also be able to return a CAR that's matching a selector). It's slightly different, however, because the user story here involves the user being willing to make iterative requests to get the data after it's local -- it doesn't require CARs; and users could keep using their existing iterative conversational API with IPFS without changing anything, while still gettting performance improvements by nudging the daemon to get all the data ready up-front.

[welcome[bot]]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[warpfork]

(I've suggested adding this to the v0.10 milestone just now, but am not sure if it will actually belong in that scope or not -- can discuss.)

[warpfork]

@oed I think I've captured some of what you've described to me as a want here -- but please correct it if I got anything wrong here.

I did generalize it a bit -- our conversation earlier started with "graphsync", but I think the description here still fits the ask and technically managed to stay one step removed from that detail.

[oed]

Thanks @warpfork, this would be a very useful feature indeed!

[BigLep]

Per 2021-07-27 go-ipfs standup conversation, this can get done once https://github.com/ipfs/go-ipfs/pull/7976 is completed

[warpfork]

A related internal doc is: https://www.notion.so/pl-strflt/IPLD-selectors-over-an-http-gateway-63cf0781ae304a3cb3f8f58b44649b03

[oed]

@warpfork is that a doc that could be shared publicly? 🙏

[stbrody]

FYI I just opened https://github.com/ipfs/go-ipfs/issues/8526 with a related feature request that would also be necessary to enable efficient traversal over complex IPLD data structures

[BigLep]

2022-01-07 discussion: a big challenge is how to put selectors on the command line, but that has been solved with Lotus so we should be able to just follow suit.