Discussion - Clients making the most of DataCap pricing power

[jnthnvctr]

From @flyworker on slack: "As a client , I not not saying that they are charging too much - it is just a discussion in which case client is willing to pay for the miners.e.g. in slingshot, I am willing to since I am eligible for the prize. But on the end user side, will I take money from the end user then I send to the miners? Then I will choose the lowest price provider, right? "

[flyworker]

One further, since miners can gain 10x or say saving 10 times cost, will the miners even give clients money offline for getting more deals? Will it became an interest exchange issue?

[jnthnvctr]

My 2c - is that this is a market - and like any others supply and demand can potentially create weird artifacts (e.g. negative oil prices earlier in the year).

My sense is that in some cases it does make sense for a miner to "pay" a client for large amounts of verified deals - but that payment might come in the form of services (i.e. handling logistics / hardware costs for data transfer, offering fast/free retrieval, etc).

I think the two levers here then that are important to consider is what is the scarcity of DataCap (relative to the network size) and mining availability.

tagging @zixuanzh for thoughts as well

[f8-ptrk]

it actually comes in the form of FIL. the price is 4FIL / 10x23GB (as by now, known sellers only - no guarantee that it is not cheaper elsewhere!)

this will be an issue until we can guarantee data caps for ~575.000 sectors each day.

[kaptinlin]

I saw in #fil-deal-market that some miners are willing to pay extra fil to receive verified deals. This should be determined by demand-supply relation. If there are miners who want to buy, can the certified clients sell it? If such exchange are not allowed, does the economic model of fil plus need to be improved? When demand is greater than supply, how to balance the market? If there is a FIP that prohibits verified clients from selling datacap, is there a mechanism to prevent them from trading privately? Do the exchange of miners and certified clients follow a free-market economy?

[NSC-FIL]

In any healthy "free" market there are still rules that clients and providers must follow. The governance of how datacap is distributed and how Filecoin+ participants (Notaries, clients, and miners) can affect the overall market for FIL is important to prevent abuse. We have to determine the ethics and rules that we as participants in this network can accept within our community. My opinion is that we need to prevent the selling of datacap to miners (even if they are willing to pay for it) and make this type of sale punishable with penalties (i.e. removing power from the miner that was found to have been bought it, black-listing the client that sold it, and if determined that a Notary was directly knowledgeable of this purchase but did not try to stop it the Notary should be penalized). My suggestion is as follows: 1) A clearly defined set of rules for all Filecoin+ participants should be posted and penalties should be shown for breaking these rules, including for the sale and purchase of datacap. 2) Complaints about potential rule breakers should be listed somewhere available for the public review. 3) An audit or public hearing should be conducted to prove with evidence the guilt or innocence of the accused. 4) If the rules were broken, the rule breakers must bear the penalties. 5) In the case of datacap, the foundation or committee that controls datacap distribution to notaries must enforce the rules In my opinion, there should be no reason for Filecoin+ to exist if we allow this type of "trading" of datacap. The purpose for Filecoin+ should be to encourage clients to store valuable data (preferably for the "good of humanity") in the Filecoin network. How valuable could this data be to the society in general or to the client for that matter, if the client is only willing to store it with storage miners that are willing to pay for it? It should be enough for clients with "verified" data to have miners setting their --verified price to 0 and storing this "valuable" data at the miners' own expense.

[f8-ptrk]

@NSC-FIL i brought up the question on slack if monetizing datacaps can coexists with the goals of storing valuable data in the sense of FIL+ or if storing data for less than no money (earning money by storing) makes this the goal of acquiring the data cap in the first place.

[flyworker]

During last governance call, we were saying that we will have the feed back from the previous clients' datacap usage before give new data cap. Did the analysis done? I see https://filecoinproject.slack.com/archives/C01DLAPKDGX/p1613017220104200 get alocation 3 times, do we have public information about how the datacap is used?

[kernelogic]

I have asked ZX when filplus notary application began: is there any monetary benefit to become a notary and the answer is no, it’s purely for the community growth sake. Now we are talking about allowing notary to sell datacap in order to make up their time? How about just give notaries a fixed amount of FIL and disallow the selling. Not everything should be sold.

In the end, it is a supply-demand issue, datacap is approved slowly, selectively and objectively. Most small miners who are also the most active on slack, who also need the datacap the most to compete with big miners, get no verified deals.

[kaptinlin]

Allowing clients to sell datacaps has the following benefits:

1. There will be more useful data stored in the filecoin network. Because notaries are more willing to allocate datacap to real application data. Give client enough incentives, and they will have the motivation to store more data in filecoin.
2. The miners will be more focused on providing storage. Because they can buy verified datacaps on the market, they can reduce their own motivation to falsify data and apply for verified data in order to obtain enough verified data to obtain more benefits.
3. Miners don't need to worry about black box operation, as long as they are willing to pay, they can buy the verified datacaps from the market.
4. Notaries work easier. They don't have to spend much times to identify whether the client is self dealing.
5. In the early filecoin network, this helped to achieve the goal of filecoin+. Let more valuable data be stored in the filecoin network.

As far as I can see, verified datacaps as an open market commodity is beneficial to all participants (notaries, clients, and miners). Only a free market can guarantee the fairness of most clients and miners. Just forbid notaries to sell datacaps and give them enough incentive.

Prevent the selling of datacap to miners has the following disadvantages: 1) There is no guarantee of fairness for miners. Is it dealing with big miners or small miners? Because storing datacap has great benefits, many miners will find ways to obtain it through illegal channels, which is unfair to honest miners. Bad money drives out good. 2) Many miners will want to become verified clients themselves instead of focusing on improving the stability of storage. And they can't really provide valuable data. 3) Notaries have to spend a lot of effort to identify these fake miners.

I've talked with many Web 2.0 companies that are willing to store valuable data in the filecoin network if there is enough incentive. If no incentive, they will continue to wait and see, or even ignore it at all. If filecoin want more valuable data, it's wise to let companies that already have a lot of valuable data start to use filecoin network instead of creating valuable data to store verified datacaps.

[NSC-FIL]

KaptinLin, your suggestions, if adopted by the community, would mean we don't need a Filecoin+ program. Just have all deals be unverified and let miners give offchain incentives to the clients who are only willing to store "valuable" data on our storage miners at the lowest price or for free. Important or useful data would have no standard, so we should not reward nor penalize nor monitor the data on the network. No governance at all.

[LandsatFile20]

As a client with verified data:

It is a free market. In order to continue to apply for datacaps, we need to make verified deals conform to the rules of Filecoin Plus.

• Valuable data & Scarce data
• long-term storage,
• Stable storage
• Store multiple copies
• Store multiple regions

[cryptowhizzard]

my 2c:

There are two discussions going on. Slingshot + verified deals and real use cases with verified deals.

For Slingshot , i agree with @LandsatFile20 , except that verified deal data should be excluded from slingshot. That data is already stored multiple times and does not need to be stored again and again. ... it is not real user case data.

For real use cases @KaptinLin is mostly right.

[f8-ptrk]

you devalue your data if you monetize your data cap @LandsatFile20 - the doubt that the data cap was applied for nothing else than monetizing it is there and will stay

[zixuanzh]

my 2c on this, there are a few concepts mixed up here and possible false claims based on incomplete information. We did discuss some of these previously on github and on govcalls and we should discuss on the next call as well. Here are some questions as I see it.

1. Notaries selling DataCap. This should be forbidden but I dont know of any proof or anything remotely close to an evidence that this is happening. Please file an issue if you see it.
2. Miners offering FIL to attract clients/developers to use their storage or clients asking for some FIL. This seems reasonable as long as it’s legitimate clients with real storage needs and follow the processes that we established (notary allocation plan). This could also be a lever to increase storage demand on Filecoin. It is also a pretty common strategy to recruit new users in other marketplaces.
3. DataCap scarcity. This is a separate question and we should diagnose the pipeline of applying for and granting DataCap for notaries. If people want to help, it will be great to understand why DataCap are not being allocated as quickly as we might have expected. Is it that there are not many clients applying? Is the process too hidden? Are notaries resource constrained?
4. Miners being notaries. This is a result of community governance and rubric and there are both large and small miners being notaries. Miners have put out some of the most comprehensive allocation plans and they have the technical experience to help onboard new clients. Miners want the ecosystem to succeed as well and there are guidelines and processes to minimize abuse. If we think we should not do this, we should discuss in the governance repo and make changes to the rubric instead of creating false claims about the process.

We all want Filecoin to be useful and successful and Filecoin Plus is a lever to do so. There have been a lot of false claims and we want to have more notaries or DataCap. However, these are somewhat dependent on us as a community figuring out the processes and tooling and building trust in the process. Constructive feedback and meaningful discussion are good but false claims will only slow us down and harm the community. Lets make it work!

[jnthnvctr]

https://github.com/filecoin-project/notary-governance/issues/66#issuecomment-777915654

Re this: it is devalued by definition (if you have to pay to acquire the DataCap then your return will be less). I think the point of contention (abstract of @LandsatFile20) is whether this is acceptable behavior for clients, and if so what is "in bounds". This is clearly unsettled and worth discussion.

Per @zixuanzh - DataCap scarcity is driving much of this - both in terms of ability for folks to receive DataCap and potentially the total amount. We should focus on debugging the root causes here.

Separately there's a question of whether the Notary did enough due diligence when awarding the DataCap (and whether proof will be posted about the legitimacy of the use case etc). The proof will be in @LandsatFile20 's deals when they make their data available.

But notably, this is a question that all early developers may run into, and there is no precedent. We are going to run into more of these issues as this program matures. Emotions are high, but we should focus on building to the right solution and debugging where things are not working - not just throwing vitriol.

And to just strongly +1 @zixuanzh 's last paragraph - our actions are going to shape the community we can attract.

[LandsatFile20]

#datacaps deals update

• Valuable data (Open dataSet) & Scarce data ( First Upload it to Filecoin )
• long-term storage Miner promise keep it more than one years
• Stable storage Miners in datacenter with 1Gbps connections +
• Store multiple copies 5 copies
• Store multiple regions French / Norway / German / NA / Singapore
• No self dealing #datacaps deals update

[NSC-FIL]

On a technical note, I was surprised that I can set my miner's --verified-price to a negative number (i.e. -0.0000000001). Does this mechanism actually work to reward datacap clients with FIL if they send a deal to my miner? If so, has anyone tested that it works correctly?
IF after our discussions and vote on this topic results in allowing clients to effectively "sell" their datacap, I would suggest we test, use, and advertise the miners' willingness to pay for datacap deals, by showing --verified-price as a negative amount. At least it would be out in the open and hopefully preserved it on-chain.

[rjan90]

@NSC-FIL No, negative prices is not working, even if you can set it as a negative number. https://filecoinproject.slack.com/archives/C01DLAPKDGX/p1608228358319200?thread_ts=1608211230.312400&cid=C01DLAPKDGX

[NSC-FIL]

Thanks rjan90, I guess the protocol wasn't originally designed with "selling" datacap as a feature. An FIP might be needed to allow it, if we think it's an acceptable practice, and we should come to a consensus on the mechanism.

[clinta]

If the market is allowed to operate this way then it should reach equilibrium when the net profit to a miner after paying for verified data is equal to the net profit from an unverified deal. If that's the kind of market that is desired then instead of rewarding a power bonus to miners with verified data, it would be simpler to just have the network pay the deal fees for verified data. Then miners don't even have to know or care whether data is verified, they set their deal fees and make a profit from deals. And the network subsidizes the cost of storing valuable data.

If the network wants to reward clients for storing valuable data, rather than just subsidizing it, that can be done too. As the client gets verified data stored, the client gets a reward. This reward can be tuned as needed to incentivize more valuable data being stored.

[f8-ptrk]

If the network wants to reward clients for storing valuable data, rather than just subsidizing it, that can be done too. As the client gets verified data stored, the client gets a reward. This reward can be tuned as needed to incentivize more valuable data being stored.

it's the notary that decides what is "valuable" data. the danger i see in the whole process (thats why i kicked of a discussion about it) is that notaries get a license to print money. it is naive to believe that an unregulated market can be tamed to exclude the notaries if we allow monetizing the rest of the verified data pipeline.

[clinta]

notaries get a license to print money

This is true, and remains true in my suggestion. But currently it is obfuscated by making the money flow from the notary to the miner (in the form of a power bonus) then back to the client. Removing that process so the notary rewards the client directly makes it more obvious what's going on and IMO would make governance easier. And it would limit the entities that need to be monitored. Currently to try and regulate this market requires regulating notaries, clients and miners. I propose removing miners from the transaction.

[zixuanzh]

notaries get a license to print money

I would like to push back on this comment. This is both incorrect and wrong and it's not how it works. There are clear guidelines and processes to minimize abuse. A comment like this demonstrates a lack of understanding. It is also not an unregulated market. There is a whole community watching and governing it.

[LandsatFile20]

##FilHub request new datacaps (pending) There must be controversy in the market (#fil-deal-market on slack). Filhub is willing to put it into practice. However, some miners are particularly disgusted with this behavior (Selling Datacaps), I think let them receive datacaps (low price or free) first. If verified clients and miners complete a large number of datacaps storage. More datacaps was stored, easier to discussions about the market. Verified client needs to store and transmit data at cost (different region), they want some profit or at least cover. Miners also want to share the revenue from datacaps .

[NSC-FIL]

Free market trading of datacap where miners end up having to paying for verified data (e.g. 10x power) on top of the high capital costs involved just to setup and maintain a mining operation, will inevidently benefit the biggest and best funded miners. This would result in a few extremely large operations left in the market. Smaller miners will quickly realize that paying clients for "kickbacks" to store their verified data will not result in enough block rewards to sustain their operations and will eventually run out of resources. Clients will expect a subsidized storage service (receive FIL or money to store verified data) because miners with more resources will pay them to grow 10x faster to maintain growth. The bigger miners who have the advantage as first-movers, consistent block rewards, and deep pockets will gladly pay for verified deals until the other miners die out and the barrier for new miners is too high. Eventually a monopoly or cartel will centralize dominance and can manipulate all aspects of the Filecoin network. This has played out time and again in industries world-wide where there was no mechanism or incentive to set anti-competitive rules and enforce them. This is why there is an Anti-trust Law in most developed countries. It is an after-thought policy in my opinion and wouldn't be necessary if proper controls to prevent collusion and corruption were put in place early on. We have a chance now to do this with Filecoin+ so I vote NOT to allow Notaries and clients to sell datacap and Not to allow miners to buy verified datacap. Consensus on enforcement is essential and must be fair and significant to deter abuse of the Filecoin+ governance process IMO.

[zixuanzh]

persisting some Slack convo here

@NSC-FIL brought up a very good point and we should be cognizant of the impact of our governance decisions. However, an important thing to point out is that compared to traditional Web2 products/projects/economies, Filecoin is a protocol. So is Filecoin Plus. And for a protocol, there could be multiple ways of interacting with it. Some of the problems that we perceive can be shaped by the immediate reality that we see (DataCaps are not being allocated as quickly as we might have expected) but some of that can just be temporary.

A concern that was brought up is about fairness and decentralization which I am a staunch supporter of. One way to achieve that could be having a community oriented Filecoin Plus Client storing public dataset with API endpoints for retrieval and distributing that allocation to small miners around the world. I recommend identifying valuable dataset as such (maybe Wikipedia to start) and applying as a Plus Client to store them. I dont see anything in our governance process that blocks it.

I am wary of straight-out forbidding clients requesting FIL/features from miners because that could be a high leverage way to bring more real clients into the Filecoin network, to overcome the onboarding cost and devote resources to improve tooling for Filecoin. As a thought experiment, we are likely quite okay with clients requesting miners to provide more features like fast retrieval, multiple copies and so on but these features are also results of dedicated resources and capital. Miners can always choose to boycott or not take the clients’ deal when the ask is outrageous.

[clinta]

I think a miner paying clients for datacap is very different than a client wanting additional features, especially as it relates to the long-term health of the network.

Clients requesting additional features are demonstrating that they care about their data and are intending to actually use the network. Letting this be part of the negotiations is good for the network because it means more real world use of filecoin.

Clients who request payment before allowing a miner to store their data are making an immediate profit just by making a deal. There is no additional incentive to actually use the network. Just pump some archive data there and make some money. There's no reason to invest any development in making maximum use of the network.

Filecoin does not want to have a reputation of being that network where miners pay clients to store data that nobody ever uses. If nobody retrieves it, verified data is no more useful to the world than the random data in committed capacity.

[jnthnvctr]

^^ Perhaps a bit of a derailment, but it would be interesting if there were a way for a client to specify something like future retrieval prices (in the vein of services Miners could offer, it could include X amt of retrievals for free)

[jnthnvctr]

Hi all - following our governance call yesterday, I wanted to follow up to push the discussion forward given there wasn't a consensus.

I thought joeg made a strong point - that outright forbidding Clients to ask miners to pay for DataCap will be difficult to monitor (there are many ways to hide or obfuscate on-chain actual payment). I believe he and Evpiole proposed a potential solution (still WIP) where we might be able to avoid this entire scenario through a different crypto economic model - which if they have a place where they're iterating I'm sure others might want to help ideate!

A few thoughts here: 1) Currently there are few ways for Clients to express preferences with Miners OUTSIDE of price of storage and asking for fast retrieval / free retrieval. Since neither of those can be enforced on chain (today) the Client is operating with trust that a miner will operate as requested. Arguably the one lever a Client has to ensure a Miner lives up to offering fast retrieval, low retrieval prices etc -> is the promise of repeat deals with a miner - but I think this set of features actually might be pretty important in the future in order to make for a better experience on both sides. This leads to one argument for Clients being able to get "paid" for storage - is that a miner can set a high price for retrieval to recoup that money, but that would really underscore the importance of making sure the Notaries are vetting properly (if a client is never going to retrieve that data, the Miner can't recoup those costs).

2) In the world of many clients, the negotiating of a power of an individual client goes down. One of the artifacts we see of the network being early, is that early adopters have substantial power (since Notaries are vetting). I think this leads to two options: a) Either we attempt to enforce a blanket rule (which to JoeG's point may be hard to enforce) b) We establish a set of norms - and leave it to the Notaries to adjudicate whether a Client is acting responsibly. I think we can shape guidelines here - but a reporting mechanism (if you see something, say something), can be used to inform any subsequent allocations a Client might want to receive.

My personal bias is towards (b) - as I think coming up with hard and fast rules that account for all the variation we'll see in the wild is hard, but I think it also pushes the onus back on the Client to act responsibly especially if they intend to make large allocations into the future. Personally am envisioning Notaries providing feedback to Clients about whats reasonable (or Miners being able to report someone acting in bad faith, and a Client having to justify their actions to the Notary). If our plan is to try and attract large volumes of Clients to the network, going with (b) would mean we wouldn't have to undo any rules in the future when the market is more mature and stable.

Curious what others thoughts are.

[NSC-FIL]

I agree with the statement that enforcing a blanket rule to restrict client actions is probably not practical and difficult to enforce so using 2(b) might be the only way to keep some limits in place. In my opinion, the situations that we need agreement on and for Notaries to monitor and limit are the following:

1) Clients that have mining operations or are inclined to use one miner should specify it in their applications. Self-dealing should be allowed (since detection is difficult otherwise) and the Notaries start off with a minimum allocation of DataCap (consensus on best cap size to deter using solely to increase power to be determined). Notaries will need guidelines on how often these clients can reapply for DataCap.

2) Clients that have applied to evenly spread out their DataCap to store with large miners should be restricted on DataCap allocation (consensus needed to set a threshold to determine at what point miners will get consistent block rewards like >300TB power) to foster decentralization by the Notaries. Close monitoring and assessing is needed before allowing them to reapply. This determination could also be based on the reputation of miners if a process is in place.

3) Clients that have applied to evenly spread out their DataCap to a combination of miners should get more DataCap since they it helps the network with decentralization. This determination could also be based on the reputation of miners if a process is in place.

4) Large projects that have been vetted by PL like Starling or Textile should get enough DataCap per their regional preference requiring coordination with Notaries in those jurisdictions. This might require more Notaries or more DataCap to be assigned to them.

5) An audit team and on-chain analytic tools should be setup either by the Filecoin Foundation or participants in the Filecoin community to ensure abuse can be detected and rules can be enforced. This should be debated by the community.

6) Clients asking for miners to pay them for verified deals (effectively selling the DataCap) should be strongly discouraged and used sparingly. But absent a way to efficiently detect and stop this from happening, the audit team or Notaries should review complaints from the community and investigate when evidence is presented.

These are some suggestions and I welcome debate on the merits of my opinions. Whether these are effective ways to promote fairness, decentralization, and useful storage on Filecoin can and should be determined by the majority of stakeholders in our community. We should set down the rules for Filecoin+ and put them into effect in the protocol soon.

[f8-ptrk]

another thing that we might want to think about is not allowing notaries to sit on data caps for long periods of time.

• we could lower the size of the data caps for notaries significantly into the lower TB range
• we could put an expiration date on the data cap a notary holds
• we could put notaries on hold if their data cap expires and let someone else handle the regional slot

i think it was @jnthnvctr who pointed out that there is plenty of data cap available with notaries but it is not gettin' handed out. we might have to survey notaries why that is or enforce "circulation"

[f8-ptrk]

maybe we can even find a way to define a metric we can enforce on chain to make data providing more power.

• data stored with a certain amount of miners
• data being received a lot
• ...

the current model gives incentives for large amounts of dead public data. as soon as the data is on chain FIL+ ends. giving incentives for long term utilization of data - independent of what that data is - might be a good idea.

why are the 100TB open source satellite photos no one ever receives over month more "useful" than the 128GB "none of your business" data that gets actively retrieved, stored all over the place etc.?

[NSC-FIL]

My 2 cents: Whatever rules we come to consensus on as a community, we should try to integrate quickly into the protocol, as much as possible to be enforced in an automated way... to remove human error as much as possible.

[dkkapur]

This topic has evolved quite a bit in the last year through learnings from experiments like bigd.exchange.

Going to close out this issue for now due to lack of active discussion.