search

ipfs / pinning-services-api-spec

Standalone, vendor-agnostic Pinning Service API for IPFS ecosystem
https://ipfs.github.io/pinning-services-api-spec/
Creative Commons Zero v1.0 Universal
100 stars 27 forks source link

Proposal: IPNS pinning #85

Open RangerMauve opened 2 years ago

RangerMauve commented 2 years ago

At the moment, I don't think IPNS is supported by some pinning services, and I'm not sure if everyone is in sync on what the behavior would actual look like.

It'd be nice if we could standardize how IPNS should be handled by pinning services (including DNSLink functionality, how updates work).

lidel commented 2 years ago

FYSA there is a proposal and discussion around "IPNS Pinning Service" at https://github.com/ipfs/devgrants/pull/142 (@mrodriguez3313) My initial idea is to add IPNS support to this spec, rhater than inventing something new. See my comment in https://github.com/ipfs/devgrants/pull/142#issuecomment-1131622631. Sanity check + feedback on that issue would be appreciated.

RangerMauve commented 2 years ago

@lidel That's great to hear. @dirkcuys and I were just talking about what to do about Milestone 5 of the Agregore Mobile devgrant and IPNS pinning. Will take a look.

mrodriguez3313 commented 2 years ago

@RangerMauve could you add a link to your project and or proposal? I would love to see the progress and roadmap

RangerMauve commented 2 years ago

@mrodriguez3313 Yup, we're posting updates about it on this github issue: https://github.com/ipfs/devgrants/pull/129

The relevant bit is Milestone 5: https://github.com/RangerMauve/devgrants/blob/ef3ac96d6aab4d498b2ecd4cd9f7d99fb29ba2a1/open-grants/open-proposal-agregore-mobile.md#milestone-5-ipfs-pinning-service-ui--integration

At the moment we're planning on supporting both the regular IPFS pinning service API for IPFS URls, and also supporting the IPFS Cluster pinning API for IPNS.

lidel commented 2 years ago

Quick brain dump on our options "how to represent /ipns/{foo}" as a CIDv1.

We need to create rules and conventions around pinning (1) cryptographic IPNS names and (2) human-readable DNSLink names.

  1. if foo is IPNS record key, then it is a CIDv1 with libp2p-key codec already
    • the main argument for not messing with existing CID is UX: user pins the {foo} from the address as-is, without the need for any conversion
  2. (open problem) find a way for pinning DNSLink names – need clarity for the future, even if we only decide to support (1).
    • Allowing DNSLink names improves UX/devexp significantly, and allows website owners to pin a DNS name once on multiple services, and then have pinning services follow it (solves cases where IPNS records are not used, or when it is ok for IPNS keys to changeover time)
    • fwiw we have ipns-ns codec (0xe5) which was proposed by ENS team but I agree with https://github.com/multiformats/multicodec/pull/136#pullrequestreview-257769901 – we should not be mixing DNSLlink and IPNS records under a single umbrella, they should have distinct codecs, if possible
    • instead, my initial idea is to add dnslink codec to remove any ambiguity – would appreciate review/sanity check in https://github.com/multiformats/multicodec/pull/268
mrodriguez3313 commented 2 years ago

Thinking about point 1) and having gone through the pinning service spec. Since, with ipns records, there are two "types" of pinning. I feel like it would make sense to differentiate that in the spec. That being Record Tracking and Record Republishing

Record Tracking, that is allowing a service to track the record you have alive Not much for the API spec would have to change here. The Pin object would hold the ipns key identifier instead of the CID and the rest would stay the same. Then the PinStatus object, would also hold the same format. The only thing I can see as possibly different would be the status results from the service. It could be queued, it could be resolving, or resolved. Tracking a record does not necesitate transferring/sharing keys at all. And you would use the endpoints for getting list of records you've pinned, posting records, and removing/stopping a record from being tracked by the service. My biggest question with this goes back to this github issue https://github.com/ipfs/go-ipfs/issues/1958#issuecomment-444201606. Does the service, as a central point, want to continue providing expired records or throw away those expired records? I have my opinion, but I'm interested to hear more people's thoughts on this? Also (slightly unrelated), how would services know how to connect to the origins to download the record/data. Is there already a library or functionality in IPFS to support this? And on the flipside how would the service distribute the content to its delegates? I know these wouldn't be specified in the spec I was just wondering for my project.

Record Republishing, that is trustlessly allowing a third party to republish your record. This is still a big dicussion, regardless I believe the general idea was/is, there being some extra key that is not a users private node key that permits republishing? And if that is the case, then nothing would have to change in the Spec in this regards either? Because then that key could go along in the Pin object in the CID? And of course a users API key or JWT access [link to that functionality](that I don't remember where I found). And if it has it's own codec then the third party service & ipfs can interpret it and allow republishing it with it?

Like @rangermauve was saying in the https://github.com/ipfs/devgrants/pull/142#issuecomment-1136437173 it would be nice to for a user to be able to specify how long they want their record to live before expiring. I just personally don't know if that would be specified in the spec? (I'm just not an architect genius, thoughts on where this would be best to have?) Maybe that could be as part of the meta field in Pin object the user submits?

Im sure you've thought of these things already @lidel @aschmahmann. Just wanted to bring them up here and see if you can offer an insight to the questions in this post or any follow up?

RangerMauve commented 2 years ago

Cross posting from another thread on encoding DNSLink domains as CIDs:

Hmmm. Personally, the CIDs for DNS Name approach seems like it'll add a lot more burden for pinning service client implementations since they'd need to include CID libraries in whatever code they're writing (I'm a dependency minimalist ðŸĪŠ).

Have you considered alternatives to CIDs like URLs? e.g. instead of passing a CID, clients could specify a ipfs://bafywhatever URL or ipns://ipfs.io/subpath URL. It'd add more work on the pinning service implementor side, but it might make the client side a lot more flexible. ðŸĪ· (I'm also really into URLs)

lidel commented 2 years ago

@mrodriguez3313 the longer we talk about this, the more sense it makes to separate "dynamic/following/tracking" pinning from "static / one time" pinning. You have good insights!

I agree with @RangerMauve that forcing DNSLink into Pin.cid field is problematic and if we could avoid that can of worms, we should.

Perhaps we should frame this work around separating static pins (/ipfs/{cid}/) from mutable ones (under /ipns/{name}/).

What if..

A separate endpoint would:

Does the service, as a central point, want to continue providing expired records or throw away those expired records? I have my opinion, but I'm interested to hear more people's thoughts on this?

My usual north star here is for people to be able to use pinning for keeping valuable datasets alive, even when the original publisher is gone.

If a user pinned an IPNS name, they expect the service to keep it alive (republish record, reprovide CIDs/data behind the record) until the pin is removed by the user. To illustrate, you don't have private IPNS keys for, but want to keep last IPNS snapshot of Wikipedia, press articles, historical scientific datasets alive.

Data (CIDs) is always reprovided. I argue that when we talk pinning, the same should be applied to IPNS record itself, even if it is expired.

Sure, the record may be expired, and sure, DHT/pubsub IPNS clients may ignore puts for it, but the pinned data should be around for clients that are ok with an older, expired record as the ultimate fallback.

how would services know how to connect to the origins to download the record/data. Is there already a library or functionality in IPFS to support this?

Pinning service takes multiaddrs from Pin.origins and preconnects to them via the equivalent of ipfs swarm connect {multiaddr}. See https://ipfs.github.io/pinning-services-api-spec/#section/Provider-hints When no origins are provided, DHT is used for content routing.

Record Republishing, that is trustlessly allowing a third party to republish your record.

Right now, AFAIK, there is no trustless way for a third party to create a new, signed IPNS record with an update expiration date. The only thing you can do in trustless fashion (without sharing private keys) is to republish old signed IPNS record. (You are free to propose a scheme for trustless publishing where publisher is only able to re-sign an existing record to bump its TTL, and is unable to change the CID it points at, but it feels like a separate research/devgrant track.)

RangerMauve commented 2 years ago

Regarding expiration I think we should encourage users to use a high value for the lifetime parameter in ipfs name publish so that we can avoid worrying about expiration for the time being (at least not as much). e.g. set it to a Year and tell users to try to be online on their original pinning device more frequenty than that. 😅 Not a perfect solution of course, but I think it could get us pretty far in terms of reliability. Might also be good to spell out the security implications of doing so.

Although I hate to admit it, I'm +1 to having an optional private key field in the new API. ðŸĪ·

Also, I'd like to point out that in the IPFS-cluster API they have two paths for adding IPFS/IPNS pins add /pins/ipfs/{path} and add /pins/ipns/{path}. This is in tandem with a add /pins/{cid} API

mrodriguez3313 commented 2 years ago

@lidel

My usual north star here is for people to be able to use pinning for keeping valuable datasets alive, even when the original publisher is gone

Yeah that is where my opinion lies as well.

periodically resolves path and pins data and IPNS record behind it – inspired by https://github.com/ipfs/go-ipfs/issues/4435

So if I understand correctly, this issue is for local development of pinning and by extension of what people want in a remote pinning service as well. But after reading it, I realized the command ipfs pin remote add --follow could be a little confusing.

I can think of two scenarios which can both be described as "following a record". The first is closer to what is being described in issue 4435. Which is that the users want to recieve updates on their own node from the original publisher (OP). Now this described in the issue is something that ideally they want baked into IPFS. Then they can circumvent any need for scripting to "follow" their content. So the command could also mean that they want to recieve updates of a certain record from the service?

The second scenario, would be to actually pin a record to the remote service (what we want to do). Where we tell a service to "follow" a record aka periodically resolve it to check back for updates. Do you think that this necessitates differentiating and adding to the IPNS spec? The first scenario is not necessarily pinning, and if it were to be included in the spec, then it seems like this functionality would have to be added into go-ipfs?? So I would think tracks would be a better word for the endpoint and functionality we are aiming for? Thoughts?

Pinning service takes multiaddrs from Pin.origins and preconnects to them via the equivalent of ipfs swarm connect {multiaddr}. See https://ipfs.github.io/pinning-services-api-spec/#section/Provider-hints

Will do! Thank you.

Right now, AFAIK, there is no trustless way ... but it feels like a separate research/devgrant track.)

Yes I agree. Definitely requires a more dedicated and focused game plan.

@RangerMauve

Regarding expiration I think we should encourage users to use a high value for the lifetime parameter in ipfs name publish

Right right I know what you mean. Since people will imagine the pinning service to be like static ones or operate them like DBs. They may forget about their records after 24-36hrs and not be online at that time and the record will expire. And since we want the default functionality to keep the expired record available. The security implications would be present and so we should lay those out for the user to educate them.

Also, I'd like to point out that in the IPFS-cluster API they have two paths for adding IPFS/IPNS pins add /pins/ipfs/{path} and add /pins/ipns/{path}. This is in tandem with a add /pins/{cid} API

Can you elaborate on this?

mrodriguez3313 commented 2 years ago

So to be a little more concrete and think publicly. It makes sense to have those added fields in the Pin object. So lets say that path, expiration, and privateKey fields get added on top of the current ones (cid,name, origins, & meta). This will give immediate access to the cid in a server response. So when records are listed via the GET /pins equivalent endpoint the user can quickly see the same data as in the current spec and the 'path' & expiration of the record. Secondly, it tells the user which cid the service has. Then, a client could double check the answer by doing a resolve on their end.

Ultimately we want to have the latest version of the record returned to us. But it may go through updates and the client may want to quickly see how many changes has the service picked up on. Since "tracking/following" is a background process that would take some time to update. The returned PinStatus response may want to include how many times the service has updated the cid a record points to in its node. So I propose adding a version or count field to PinStatus.

Basically we want similar results like ipfs name resolve <name> from the service. So having the cid along with path will help understand results (and submissions) easier. Thoughts?

Edit: 9/7/22