jimmylee
commented
4 years ago Question & Answer with @whyrusleeping and a user on Filecoin Public Slack
Hey all, last time I looked at the filecoin project, it was pretty clear on what it did and how it did it. However, with the latest website change (and docs) I feel like a lot of details that were clear to me were left out, and I'm afraid that those specific implementations aren't what they used to be. Here are some specific questions:
To store data, I am not anymore making a deal with a market, but between induvidual "local" nodes that all offer prices to me? How exactly could someone get an audit on global/market-wide pricing quickly enough to make a decision and induce that hyper-competitiveness if nearby nodes could "crowd out" smaller and less-reachable ones with higher prices? After I made a deal with a miner, and sent my data to them, how exactly is the data supposed to be recoverable and healable if that miner goes down? How exactly does the self-healing aspect of this network come into play if I make a deal with an individual miner? How do I know that whatever miner "takes the burden" would not immediately start asking me ridiculously high prices for retrieval?
How exactly would I retrieve my data as an end-user? I'm referring to the fact that if someone on phone decided to download high-res images they made by themselves, how could the price be determined? If the "bus factor" of the data is 1 (all miners controlled by a single party), how would that stop them racking up off-chain prices of retrieval? How exactly would "private data" come into play here? If I store my data on the network, am I responsible for securing it so that somebody else cannot just request the CID and start reading out my data? Where falls the burden of verification/authentication?
The site mentions Dapps, I personally have not yet really dug into these, but how exactly can a website app that wants to be free for users (like modern websites today) be free on a network where retrieval is monetary? What has an application developer to gain for putting their Dapp on the filecoin network? In my eyes, they'd not get any money off of it, and the users would need to pay to use them, which isn't exactly a strong incentive to use it.
These are great questions, and point out a bit the boundaries of what filecoin is today, and what features will come after launch.
Miner pricing is communicated via signed 'storage asks'. Currently, the software just queries a miner directly for their asks, but thats not necessary. An aggregator service can query asks from a large number of miners, and present those quickly to clients, so clients can ask an aggregator for the best set of miners. Since the asks are signed, and have temporal validity information baked in, clients can trust asks received from anyone
Automatic repair of faulted data is a feature we've pushed off until after mainnet launch. For now, the way to ensure resiliency is to store your data with multiple miners, to gain some level of redundancy. If you want to learn more about how we are thinking about repair in the future, here are some notes: https://github.com/filecoin-project/specs/pull/245/files Pricing for retrieval in an efficient market should approach a small premium over the cost of bandwidth. Miners who want to make more money shouldnt turn away the opportunity to make more money, as long as the proposition is profitable. To avoid extortion, always ensure you store your data with a fairly decentralized set of miners (and note: its pretty difficult for a miner to be sure they are the only person storing a particular piece of data, especially if you encrypt the data) Miners currently provide a 'dumb box' interface and will serve anyone any data they have. Maybe in the future miners will offer ACLs and logins and such, but that requires that you trust the miner. The recommended (and safest) approach here is to encrypt data you don't want others to see yourself, before storing it.
There are a ton of potential applications that can be built on decentralized file storage, but going down your line of questioning here, There are a good number of websites out there right now where file downloads are supported entirely through huge amounts of ads (the sort of thing where you click through three pages of ads before you get to your really slow download). All of those types of things could be easily replaced a filecoin incentivized retrieval setup, paying small amounts of filecoin directly for whatever files you were hoping to download (the biggest one here in my mind is when youre trying to download mods for games). There are also a bunch of large datasets out there that are hosted through amazons 'pay per download' s3 buckets, filecoin retrieval easily competes with that too. And more generally, More Dapps will become possible once we allow for arbitrary user programmed actors (smart contracts) because at that point, any dapp that works on any other platform can work on filecoin, with the added bonus of having storage so nearby.
did not know about repair miners, this is a really interesting feature which laid this concern to rest, from my previous knowledge about ethereum contracts, i can guess how this is supposed to work in-network, and how failover could occur entirely within the logic of the network. (Would this mean that the repair miner has a "budget" that it can spend on delegating new storage miners when previous ones fail? Or are tokens "refunded" when the original storage miner fails, and the repair miner just reallocates it as the budget for a new storage miner contract?)
Yeah, pretty much the idea there is that the repair miner would charge a premium on the underlying storage, assuming some failure rate, and would be responsible for the extra costs if they chose unreliable storage miners to ultimately store the data
We descoped the repair miner work for mainnet launch as the network is complicated enough without it, and its relatively straightforward as an upgrade later
I don't feel like my concern behind no.5 has been specifically addressed; how could filecoin provide (to all parties) a more alluring setup? The truth of the matter is right now that internet users enjoy incredible freedom and free content for "nuisances" in the form of ads, from my perspective it's not exactly alluring that to download a file, even with the help of extensions or locally-started clients to help a stunted browser with the actual retrieval, you'd have to "pay somebody", in an era where everything is for "free".
I'm specifically asking how filecoin is supposed to allure to app developers and users to actually work as a service that's either as or more "pleasant" to use, if a user has a transparent wallet built into their browser and gets this into their routine and idea of what's acceptable, then yeah it can take off, but that needs to be a huge shift in how you think about the internet.
(nothing on the internet is free, ads make you the product, free websites take your data to sell to the highest bidder, but nobody on the internet understands this fundamentally, and so they'd not like it if something suddenly becomes "not free", even in small amounts) (edited)
Yeah, I tend to agree on the point youre making here. I think the primary usecase for filecoin at first will be more 'long term' storage applications. Archival, personal backups, etc. As well as applications like existing dapps that don't want to (or 'cant') store their data in traditional ways. There is fairly significant demand among current decentralized applications for a solution to how their data should be stored, and a good number of them already use ipfs (but resort to pinning data themselves, on their servers to ensure the data sticks around). These apps already have wallets built in usually, and adding some sort of micropayment for the data they use isnt too big of a stretch. On the point of 'how could Filecoin provide a more alluring setup' to existing non-decentralized applications, I agree that it's going to require a pretty major shift in how we think about the internet. But at the same time, I am pretty excited for such a shift, and I think that things are heading that way (slowly) in general. Browser vendors and major tech companies are getting more into decentralized tech and crypto (see Brave, Opera, and Firefox's dweb experiments)
It also requires a major shift in how we see ourselves in relation to the internet, "give a little, take a little" needs to get back into the public consciousness, no matter for what, for this to even become "normal". You mention persistent storage for Dapps, you're right that that is actually a very good and applicable usecase to this (instead of having a RESTful api store the data somewhere anyway). Maybe the adopters and users would be the ones who actually care about their privacy, and see this as a reasonable tradeoff to be 100% sure that data stays in their hands, they can even turn off REST that way if it means that everything happens in a self-contained environment, where data is only edited from a personal private key, from a distributed storage, on a distributed app where no one person really hosts it, or holds the keys for it. One last question, how would updating data work with filecoin? If someone would use a Dapp, and they'd update their private data (program data, settings data), how'd that be effectively updated? Would partial updates be allowed to occur on miners? Transparently?
Maybe the adopters and users would be the ones who actually care about their privacy
I think we're gonna get a lot of these yeah, and its a very good userbase for us to grow.
how would updating data work with filecoin?
So this is ugly right now, we have some really good ideas around 'warm' storage (that is mutable, and provable) that we will probably implement in the near future, but for now, your app will have to treat filecoin as an append-only log. If you want to change your data, you just write new data.
The 'warm' storage thing can be done with a small amount of trust , where you make a deal with a miner with a start date quite far in the future. The miner can choose to store your data in a sector now (but they won't get paid for proving it until the actual start date) or they can hold it for you (and even send you proofs of it on request) and you can then send them new data to overwrite it, along with a new storage deal that overwrites the previous one.
theres a pretty large design space here, and we can do a bunch of different things depending on the levels of trust involved, the price sensetivity, and the frequency of updates clients desire
angiemaguire
Introduction
These are snippets I've collected while monitoring the Filecoin network communities.
Reading List
https://beta.spec.filecoin.io/