search

textileio / broker-core

Broker for the Filecoin network
MIT License
4 stars 1 forks source link

Enable status by CID #57

Open andrewxhill opened 3 years ago

andrewxhill commented 3 years ago

Right now we are using the unique request id for checking status. This has the downside of forcing dapps to maintain some state about data stored on top of the data stored.

I was wondering, why not change it to just check by CID. perhaps it could return the life-time information about deals containing a CID? a dapp could then even verify that, if a cid is stored, the expiration is sufficiently far in the future to maybe not even store again.

andrewxhill commented 3 years ago

if too costly to maintain the lifetime map of a cid, it could simply return the latest storage status. that should be a direct swap out for requestid.

asutula commented 3 years ago

This is a deep one. Might warrant some higher level conversation about the design of the broker and what services it provides and how those services work. Again, @jsign will have some input here, related to both the current broker design and functionality as well as whatever he things about future directions. Some thoughts/questions that come to mind for me:

What is the service we want to provide? A highly configurable storage api that clients can use to store their data to their exact specifications (a la Powergate), or a dead simple global datastore backed by filecoin? The answer to that question will determine the outcome of the issues/questions raised in your second paragraph. I personally think providing the "highly configurable storage api" will be more useful to users in the long run, but definitely open to discussion and ideas there.

About your first paragraph, using the unique request id for checking status... I agree that isn't ideal. I'm wondering if there is a way around that no matter the answer to the "highly configurable storage api vs dead simple global datastore" question is. For example, can we say that for a given public key or client/broker session, that a cid is unique within that scope? (it is already), and then just use the public key/session to scope status requests by cid to the public key/session <-> cid unique combination we need to return the status for? Seems like that could work.

asutula commented 3 years ago

Even though I'm currently thinking "the highly configurable storage api will be more useful to users", the wheels in my head are spinning while thinking about what future storage apis and markets look like. This might be like post tokenization of Yobi, but ideas about something that looks more like a global data store that is maintained like a DAO... Data continuing to be stored or dropped from storage based on support or lack thereof from the network, etc, etc. In that way, a global data store sounds really interesting.

asutula commented 3 years ago

From the product call today, a couple bits:

@jsign said it would be possible to avoid using random ids correlated to cids. This could be done by making a composite id from the user public key + cid or something like that.

@andrewxhill made the good point that we are in control of all the assumptions for now, so making some very simple rules might be a good way to start. For example, a cid can only be stored once per provider. If another user tries to store the same cid, we just return the current/already stored status and storage info. In this case, the id would just be the cid, no need to correlate with any public key or use random ids. Of course, we'd want to be careful about locking ourselves into a inflexible situation in future if/when new requirements emerge.