Modification: LDN Rules- Improve the basic rules for private data application

[Sunnyiscoming]

Issue Description

The FIL+ E project is still in the stage of communication, and clients with private data want to go aboarding in the LDN (https://github.com/filecoin-project/filecoin-plus-large-datasets/issues) . It is suggested to improve the basic rules for private data application and obtain more community consensus. The number of notaries required in the current private data request review criteria is relatively small, which has resulted in a lack of consensus on the level of trust in the content stored within the project. In the second notary period, the support of 5 notaries (about 25% of the total) was very reasonable. However, as the number of notaries increases, the requirement for the number of notaries needs to be re-discussed and agreed upon.

Impact

1. Follow the clients' request for data privacy and improve clients' satisfaction.
2. More efficient granting of Datacap and growth of community consensus.
3. Clarify the basic rules for private data in the LDN (https://github.com/filecoin-project/filecoin-plus-large-datasets/issues)

Proposed Solution(s)

1. Consensus of notaries

Raise and standardize the community consensus threshold from "at least 4~5 notaries from 3 different regions" to "at least 12 notaries from 3 different regions".

• 12 notaries represent around 25% of the currently active notaries
• 3 different regions represent at least 60% regions covered

2. Storage Provider Requirements

• The SPs shall possess of the sector pledge of 10 times Datacap requested in the application.

• Requirements for the number of nodes that apply for different amount of Datacap

  • 500T≤DataCap requested≤4P -- the number of nodes ≥5
  • 4P < DataCap requested≤9P -- the number of nodes ≥10
  • 9P < DataCap requested -- the number of nodes ≥13

3. Data Samples Requirements

• The applicant should provide data samples for the notaries to view.

Timeline

The community reached a consensus

Technical dependencies

N/A

End of POC checkpoint (if applicable)

N/A

Risks and mitigations

If it decrease the overall experience, we should not proceed with this change.

Related Issues

[DataCap Application] USC Shoah Foundation · Issue #53 · filecoin-project/filecoin-plus-large-datasets · GitHub Victor Chang Cardiac Research Institute · Issue #425 · filecoin-project/filecoin-plus-large-datasets · GitHub Proposal: Project Antarctic - 10 PiB Data Set / 50 PiB DataCap · Issue #489 · filecoin-project/notary-governance · GitHub Proposal: Project Beacon - 11.8 PiB Data Set / 60 PiB DataCap · Issue #564 · filecoin-project/notary-governance · GitHub Proposal: The New Ads - 1 PiB Dataset / 5 PiB Datacap · Issue #573 · filecoin-project/notary-governance · GitHub Proposal: Spexigeo · Issue #574 · filecoin-project/notary-governance · GitHub

[Carohere]

I'm glad that someone in the community has finally noticed the recent chaos. The bar is literally too low that that only four notaries are needed to get through a private data application.

For example, as the SP and notary for Antarctica - HOLON has repeatedly granted DCs to other SPs working on the same project with them and in this case only 3 other notaries are needed to simply play the system...I have serious doubts about their credibility, this whole sp/client-notary relationship just shouldn't exist at all.

I believe that increasing the number of notaries is an effective measure to protect filecoin's reputation as well as to maintain a sustainable ecosystem.

[Carohere]

And i would like add some of my thoughts based on this proposal.

Issue Description

There have been a lot of private data applications in the community recently, but every single one of them has different rules applied. We need to address the current inefficiencies and confusion, as well as maintain the same criteria for all applications in the spirit of web3 fairness. It is essential to discuss and agree on a set of standards that will work prior to Fil-E implementation.

Motivation

As a platform for data storage, Filecoin needs to welcome and embrace the need for more real-world data storage to better achieve the final goal of Filecoin.

Proposed Solutions

• Besides what sunnyiscoming proposed, to better regulate the allocation plan I think there should be a limit on the maximum allocation of nodes where no more than 25% of the total data can be stored with any one storage provider.

• To better avoid confusion & improve efficiency, I believe that the community should support and provide services as long as the following two points are met: 1) The notary should still perform due diligence and proceed to implement validation of the clients and their data storage requirements. 2) The project should be supported by a sufficient number of notaries as mentioned upon.

Impact

1. Follow the clients' request for data privacy and improve clients' satisfaction
2. More efficient granting of Datacap and growth of community consensus

[dannyob]

So, I'm warm to the idea of requiring 75% of the data be stored across SPs (although I'd like to dive a bit deeper into how we actually limit collusion here: I'd rather that we specified a wide distribution across the Fil+E program as a whole, for instance.)

I don't really think that 12 notaries makes sense, because I'm not sure what it buys us. Also my experience with consensus-driven mechanisms (cough, noisebridge) convinces me that as the number required goes up, the probability of actually passing anything collapses rapidly to zero. Maybe if you had it as a majority vote?

[MetaWaveInfo]

The current private data validation has too much centralized power for notaries, and the number is too small to represent notaries as a whole. Increasing the number of notaries would be more conducive to consensus, but 25% is not really enough, we suggest raising it to 15 notaries.

[Carohere]

Hey @dannyob, would you like to share more about how you think collusion can be effectively limited?

The reason why this does make sense for me is quite simple. I see the role of notary as a representative of the community who exercises discretion on whether an application is compliant or not. The fact of being a Notary is an official certification of one' s reputation and the confidence that s/he will comply with transparency. But from what i saw, 4-5 notaries are no longer capable of representing the notary public or the officials since some of them are directly involved in projects that they participate as SPs. 4-1=3, no doubt this has further lowered the criteria for application approval and makes their decisions less convincing. And that's exactly why i think this has to be changed.

[1475Notary]

I think it's essential to place limits on the maximum storage percentage of nodes. In addition, we suggest suspending the release of DC immediately if the limits are not followed. Some notaries are ignoring these requirements recently, and we are keen to have a clear set of enforceable rules released ASAP!

[Holiday507]

From 4 or 5 to 12 may seem like a significant increase in figures, but we should be aware of the fact that the number of notaries has increased tremendously compared to the v2 period. From this perspective, 12 notaries which is about 25% is a very appropriate ratio. We are very much in favor of this modification! In addition, I think the maximum storage ratio of a single node is also worth paying attention to. To a certain extent, self-dealing can be avoided.

[kevzak]

@Carohere - I'm curious on your comment: "no more than 25% of the total data can be stored with any one storage provider"

We are actively discussing the Lead SP role in Enterprise projects. LINK

Would this rule you are proposing apply for any project? And 25% means of 5 copies total (for example)? Or 25% of one copy?

[graceonline9]

While I believe that notaries will keep the promises they make in their applications, we should also refine the standards to avoid collusion. I don't see any harm in improving the standards.

[Destore2023]

Quote the understanding of Deep as a comment on it. Also ByteBase gives support to the three points of this proposal and would like to help integrate them into E-FIL rules. @Sunnyiscoming

[Carohere]

@kevzak I meant 25% of 5 copies in total. Not sure if you were at the last notary governance call, back then your colleague had clearly said that 4 SPs and max of 30% storage were only the suggestions given by the official. Actually, I like the idea that the node ratio limit should be chosen by the client and then monitored all the way through the notary's due diligence to ensure it is reasonable. This is very feasible in theory, but from what I saw, with fairness already greatly hampered by the notary's self-involvement, I don't think this would be nearly as effective as what you guys expected or assumed.

More importantly, I don't see how Filecoin can achieve the most fundamental goal of decentralized storage without limiting the allocation ratio. And I believe none of us would like to see DC being stored on only one single node, all in one node may sound unrealistic today, but it is something that could happen in the near future... so my advice is to change it now before it's too late.

BTW I'm not around here that often, if there's anything that you would like to discuss, feel free to reach me on slack.

[cryptowhizzard]

I would like to emphasis something here.

Reading through this topic i notice that some of you have trouble with some notary’s decisions.

The other side of this story is that a notary cannot be a DP either, hence they cannot be involved in anything in our space to make clean decisions, meaning no-one can be a candidate.

Second, people working with a company acting as SP or DP generally do know what they talk about and what they need to do. When during start there is a learning curve or mistakes are made please see this as a learning moment and not as intentional unless repeated after explanation. Doing duedilligence, looking at performance of DP’s and SP’s, checking if everything is retrievable, checking if ratio’s are met, checking if the data is as is … it is a hell of a “unpaid” job on voluntary bases next to our busy lives.

[f8-ptrk]

you singed up for it - that that work would need to be done was clear in the beginning, otherwise we can just approve everyone

where we need to go with this in my eyes is a point where clients, data prepares, name it how you want present their plan/distribution schedule before hand and then we can integrate that with something like evergreen for deal distribution, setup automated checking with the fvm etc. - in the end the notaries will get less flack then now as they just need to make a decision if a plan fits the data attached to it and then automated systems take care of the plan being implemented.

example:

• 1Pb of whatnot-data
• 5 copies
• miners on 3 continents fitting pattern "high quality"
• exec time 3 month

from there evergreen takes either predefined miners or the ones it knows to fit the patterns and offers the deals to them. then we have a fvm contract that checks if we are on schedule for the 3 month, if all miners get data, no one more than 30% and so on - then auto renews the datacap - or not!

evergreen has automated retrieval testing etc. build in! it's a great thing and the benefit to notaries, deals, FIL+ of using it would be great (and yes it will be ready to do that soon)

[edit]

code is law - we need to go there sooner than later. get as much of the notaries work to the fvm. we do not need notaries to check stuff as soon we can properly code things into the fvm. the rules need to be clear, sure - then it will be "easy" to go make this an automated success story - now FIL+ is just mayhem, fraud and a liability to the ecosystem.

[cryptowhizzard]

Don’t get me wrong, i do agree with you. I just want to give a message that a notary “mistake” is not always intentional. If we punish mistakes we go on the wrong path. We should make clearer rules and focus on prevention imho, b/c without some level of thrust this planet cannot survive.

[Carohere]

Hi @cryptowhizzard, I think I'm one of those who " have trouble with some notary's decisions " you're talking about and i would like to share some of my thoughts.

The other side of this story is that a notary cannot be a DP either, hence they cannot be involved in anything in our space to make clean decisions, meaning no-one can be a candidate.

I didn't get what you mean by this, could you explain in more detail?

Second, people working with a company acting as SP or DP generally do know what they talk about and what they need to do. When during start there is a learning curve or mistakes are made please see this as a learning moment and not as intentional unless repeated after explanation.

True, no one is perfect and it's normal to make mistakes in the learning process, but I don't think it's appropriate to not give any explanation for the problems that have been pointed out. I'm not targeting any specific case, all I'm looking for is to understand what people, other community members think about this type of behavior and the ultimate goal is to ascertain whether changes should be made to avoid more of these behaviors in the future.

By the way, I think it is unreasonable that there are currently no restrictions on the participation of individuals/organizations who are both notaries and SPs in projects that they are involved in. Self-avoidance is definitely something we need to focus on in order to achieve a highly transparent and balanced ecosystem. But this is not the subject we are talking about right now, and I won't go further on this. Just in case you're interested, Im working on a proposal on self-avoidance for notaries and probably will submit it during the week, you're more than welcome to join the discussion.

Doing duedilligence, looking at performance of DP’s and SP’s, checking if everything is retrievable, checking if ratio’s are met, checking if the data is as is … it is a hell of a “unpaid” job on voluntary bases next to our busy lives.

Appreciate your hard work, but I strongly disagree with your point of unpaid volunteer work. It is true that on a monetary level it is unpaid, but the high recognition you receive in terms of reputation is definitely a positive impact either on your cooperation with other SPs or on your DC applications, it is absolutely beneficial for you even if it's indriectly related.

There are very, very few people in the world who are great enough to do completely reward-free work, and I'm sure there are even fewer among the participants of Filecoin or at least I'm not. It's natural for people to participate for their own or business interests, and there is nothing to be ashamed of. I am in the same boat, the main reason I am here is to keep up with the community trend so i can better invest in my fil.

Lastly, no one is being forced to become a notary, it's purely a personal choice. The responsibilities are clearly stated in the notary application, and all those who participated in the election made clear the amount of time they are willing to devote to Filecoin each week and their commitment to abide by the community rules. If anyone can't keep their word, that's their own problem. And I have also noticed that you were notary in v2, if this is that much of a burden for you, I suggest you take a break in the next election. Like personally, I'm happy to do this pro bono work, you know as someone with no tech background, no storage needs, this is pretty much the only role I could be involved with filecoin other than being a community member, but unfortunately, I don't have the opportunity. And I'm pretty confident that I'm not the only one who thinks this way.

[Carohere]

There is not yet a good consensus on encrypted data applications in LDN. Like exactly how many notaries are required to approve the application? How can we better verify the authenticity of the data? How to better ensure the decentralization of data by the number of nodes? I believe no one here wants to see more confusion after the DC is granted, and even worse any remedy will be limited by then.

I think this proposal can greatly enhance the authenticity of private data and application through a more improved standard, but unfortunately, there is not a lot of active participation seen here.

@Kevin-FF-USA I'd like to take this to the governance meeting this week for discussion if time allows.

I don't think we should move forward with any application until we have reached a consensus and a uniform standard in place. @dkkapur @raghavrmadya would love to have your opinions here, thanks!

And here are the encrypted data requests in progress I found, may not be a complete list, please feel free to add any.

564 Project Beacon

573 The New Ads

574 Spexigeo

601 Global Computer

[Carohere]

596

[raghavrmadya]

@Sunnyiscoming thank you for the proposal. I've been silently following this discussion over the past few days and taken the time to organize some thoughts before pitching in. Here's what I think -

Increasing the consensus threshold does not necessarily mean increased trust and transparency.

So, I'm warm to the idea of requiring 75% of the data be stored across SPs (although I'd like to dive a bit deeper into how we actually limit collusion here: I'd rather that we specified a wide distribution across the Fil+E program as a whole, for instance.)

I don't really think that 12 notaries makes sense, because I'm not sure what it buys us. Also my experience with consensus-driven mechanisms (cough, noisebridge) convinces me that as the number required goes up, the probability of actually passing anything collapses rapidly to zero. Maybe if you had it as a majority vote?

So, I'm warm to the idea of requiring 75% of the data be stored across SPs (although I'd like to dive a bit deeper into how we actually limit collusion here: I'd rather that we specified a wide distribution across the Fil+E program as a whole, for instance.)

I don't really think that 12 notaries makes sense, because I'm not sure what it buys us. Also my experience with consensus-driven mechanisms (cough, noisebridge) convinces me that as the number required goes up, the probability of actually passing anything collapses rapidly to zero. Maybe if you had it as a majority vote?

I do align with this.

@Carohere - "The fact of being a Notary is an official certification of one' s reputation and the confidence that s/he will comply with transparency". I would like to sway away from anything "official". At the end of the day, we want everything to be community driven and I don't think being a notary necessarily guarantees compliance

Instead of blocking current private/encrypted data applications or increasing the number of eyes, we should push for more transparency on how and why the 4-5 notaries who have supported the application came to their decision. I've seen examples where clients have opened notary governance issues and notaries have provided justification for their support (example - https://github.com/filecoin-project/notary-governance/issues/573)

I think more proposals such as #593 from the community can be helpful where we focus on the quality, not the quantity of due diligence. This also creates more accountability for notaries who do support an application.

I understand that automatically punishing notary mistakes is not the right way @cryptowhizzard but I do think if we notice trends of mistakes that make us believe they were intentional, penalties must be imposed. Knowing why a notary supported a certain application and not another one can help us get there.

On the second point of SP distribution requirements, I have similar opinions. More nodes for more data do not guarantee safety. Given the state of retrievals on the network today, private clients might be hesitant to store with the proposed number of nodes. That said, I do agree a floor (minimum threshold) of SP distribution is needed and I don't think it needs to be different than the current requirements.

3. Making data samples viewing and singing NDAs is just not sustainable and is not a great long-term plan. I know clients have done this in the past as an experiment and it has worked but I'm hesitant to make this the standard. Notaries can do KYC and assess client trust in many other ways which is why they were elected in the first place. If they are unsure, they have other notaries and the community to support them in the due diligence process. Saying no is always an option.

TLDR - Quality of judgment and due diligence over quantity. Increase accountability instead of thresholds.

Welcome opinions, agreements/disagreements, and thoughts.

[Carohere]

Hi @raghavrmadya, I disagree. It's true that filecoin values community participation over the traditional world and tbh that's the reason why I like here. But community-driven is not the equivalent of community-determined. Regardless of how we define it, the final decision of applications depends on the "official" and by official I mean due diligence by notaries + final decision by PL.

And yes, becoming a notary does not necessarily guarantee compliance. I agree. Despite their commitment, in many cases they can't keep their words whether driven by interests or personal relations. And that doesn't negate the fact that the scoring scheme of notary election is designed to select trustworthy candidates, otherwise everyone could be a notary.

I think the compliance of application is directly related to due diligence. I would say the main reason for various disputes is caused by the low credibility of notaries' judgments. How to improve, to better motivate notaries to strictly comply with their commitments is definitely a part we need to focus on. Enhancing the quality of due diligence is a good idea. But this part can't be done overnight. We can open a discussion about it, to figure it out thoroughly with the help from community members. But it sure takes a lot time to get ppl involved. And until we get a complete modification on that we should not ignore the chaos in the community atm. What we need is a measure that can be implemented right away. I think increasing the number of notaries would be perfect to reach our goal. If the data is genuine I don't know any applications where 12 notaries would be a hindrance. Plus the constraints of geography from three continents will also prevent or at least greatly reduce the possibility of collusion.

As for #573 that you mentinoed I don't think it should be used as a template for compliance. I'm like sure that cryptowhizzard is either working or even running Dcent. https://dcent.nl/about/ And he actually disclosed it in the notary application, appreciate the honesty. But with the direct interest tied I don't think his support is any more persuasive. And as I have introduced in the last governance call, this's also why I think notaries should not serve multiple roles in one application. BTW would love to have your opinion on that as well if you get time.

About point 3, I don't know if you noticed that this proposal targets on private data requests. With more notaries being included I think due diligence under NDAs is enforceable. How else can applications involving private data assuage clients' concerns? I don't have any better idea in mind. But if you have or anyone else has a better solution, I would love to know.

If they are unsure, they have other notaries and the community to support them in the due diligence process. Saying no is always an option.

I don't think i get what you mean by this, could you explain further?

Anyway in a nutshell I think we are on the same page that the ultimate goal of all this is to ensure the compliance and credibility of applications. And in my opinion the relation between the quality and quantity of notaries is reciprocal.

1) In high-quality cases, there is no need to increase the quantity (but that is clearly not the scenario we are facing rn) 2) In the case of no high/good-quality, the quantity should be PRIORITY. 3) Only quality and quantity in parallel will be the most perfect solution in the future.

[Chris00618]

There are lots of proposals recently. Why can't we use voting tool to make quick decisions and let the REAL clients participate in making rules related to themselves? This can reflect the spirit of web3.

[bjchend]

I would like to elaborate on my point of view. It's OK to raise the threshold for review and verification, but there needs to be a limit. Don't forget the ultimate goal of LDN. Everything is about improving the Fil ecosystem and getting more customers and more applications to use Filcoin, not setting more thresholds.