filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale

110 stars 62 forks source link

[DataCap Application]FileDrive Datasets #78

Closed laurarenpanda closed 2 years ago

laurarenpanda commented 2 years ago

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

Organization Name: FileDrive
Website / Social Media: https://twitter.com/FileDrive1
Total amount of DataCap being requested (between 500 TiB and 5 PiB): 5 PiB
Weekly allocation of DataCap requested (usually between 1-100TiB): 250 TiB
On-chain address for first allocation: f3wgfwtrs5p6jrkwfl2mksqa2ivgbgdjjrhjbefy3n7qzvotc3y6sazmp5gfyj7um6jlgdvlbiepzawnc6wxtq

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Share a brief history of your project and organization.

Our project: FileDrive Datasets

FileDrive Datasets is a platform to effectively connect the huge storage market that Filecoin has built with publishers of public datasets.
The Filecoin network provides reliable, secure, and affordable decentralized storage services, and Filedrive Team wants to deliver these benefits to end-users by building a public dataset platform.
It is challenging to attract traditional Cloud Storage and Object-base Storage users to the Filecoin network and benefit from it. Developers in the Felicoin ecosystem, such as FileDrive, need to face this challenge together.
As a member of the Filecoin ecosystem, FileDrive Team has been insisting on developing useful tools to make it easier for users to store their data onto the Filecoin network. 

FileDrive Datasets has integrated a group of tools developed by FileDrive Team to provide storage service with the compatibility of both Cloud Storage and Object-base Storage and better user experience to attract more users.
The users of FileDrive Datasets are public datasets' publishers and managers and anyone who demands public data.
This platform will allow anyone to access data they are interested in or publish, manage and maintain their datasets. A pre-review is necessary before users get permission to log in FileDrive Datasets' system to ensure their datasets are with the prevision of FileDrive Datasets' rules and local laws.
After datasets have been uploaded onto the platform, these data will be aggregated or split into data pieces in specific sizes by Go-Graphsplit, and then stored onto the Filecoin network under the cooperation with Filecoin storage providers.

Our team: FileDrive

FileDrive Team has always defined ourselves as tool developers and infrastructure builders in the Filecoin ecosystem. From 2019, we continuously focus on technical solutions and development based on IPFS protocol and the Filecoin network and do our best to contribute to the community.
Over 80% of our team are qualified engineers, and half of them have more than 10-year development experience in multiple industries, including Communication, the Internet, and blockchain.
Since last year, we have participated in Slingshot Competition, become one of the top teams, and stored over 5 PiB useful data from public datasets to the Filecoin network.
To contribute to the Filecoin Community, we developed an open-source data prep tool Graphsplit and FIL+ project dashboard filplus.info.

What is the primary source of funding for this project?

FileDriven Team, rewards from the Slingshot Competition, Filecoin DevGrants, Mircogrants and a series of Hackathons.

What other projects/ecosystem stakeholders is this project associated with?

FileDrive Dataset is an open dataset platform on IPFS Network, and all data will store on Filecoin Network through the Filecoin Plus project. Since that the primary ecosystem stakeholders are IPFS and Filecoin.

Use-case details

Describe the data being stored onto Filecoin

The data being stored onto Filecoin is from publicly available datasets in various academic disciplines, including humanity, nature science, formal science, etc.

Where was the data in this dataset sourced from?

All data is from public open datasets.

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this.

FileDrive Dataset: https://datasets.filedrive.io/

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

Yes, it is. All data can be retrieved by anyone on both IPFS and Filecoin Network.

What is the expected retrieval frequency for this data?

The data of FileDrive Dataset will all be pinned on IPFS Network first, using FileDAG powered by FileDrive, before being stored on Filecoin, which means users could mainly have two different ways to retrieve data, thought IPFS or Filecoin.
So the retrieval frequency depends on users' needs.
Our team will check on a regular basis to ensure each way is available for data retrieval.

For how long do you plan to keep this dataset stored on Filecoin?

This data will be stored for at least 1 year on Filecoin, so the verified deals will use a 1-year minimum deal duration.
Ideally, this project will be a permanent archival on the Filecoin network, as long as there are actual users and data requirements.

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

All regions, as long as data transmission can be stable and successful.

How will you be distributing your data to storage providers? Is there an offline data transfer process?

Mainly thought online storage deals. Offline data transfer might be considered in some certain situations.
The expected data onboarding rate is 250TiB per week, which is around 35TiB per day. 
However, it will be influenced by factors, such as the speed of data transmission(especially inter-regional transmission), daily power growth of storage providers, base fee, equipment performance, etc. For these reasons, the actual data onboarding rate may differ from our expectations.

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

Major characteristics about choosing the storage providers:
- location
- transmission speed
- deal success rate
- previous experience of real data storage
- stability of their nodes

How will you be distributing deals across storage providers?

All storage deals will be verified if they have enough FIL to pledge and their equipment can handle.

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

Yes, we do.
Suggestions and feedbacks can help us optimate this project and cooperate with more great storage providers from all over the world.

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Request number 7

Multisig Notary address

f01403243

Client address

f3wgfwtrs5p6jrkwfl2mksqa2ivgbgdjjrhjbefy3n7qzvotc3y6sazmp5gfyj7um6jlgdvlbiepzawnc6wxtq

DataCap allocation requested

4.20TiB

large-datacap-requests[bot] commented 2 years ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01403243

Client address

f3wgfwtrs5p6jrkwfl2mksqa2ivgbgdjjrhjbefy3n7qzvotc3y6sazmp5gfyj7um6jlgdvlbiepzawnc6wxtq

Last two approvers

cryptowhizzard & MegTei

Rule to calculate the allocation request amount

800% of weekly dc amount requested

DataCap allocation requested

4.20TiB

Total DataCap granted for client so far

4.99PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

4.20TiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
154526	38	1.70PiB	12.52	401.77TiB

IreneYoung commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebcnqj64k7ia35fg7aolccewmiiguxwfwjfunwc6gynaxpdl766pa

Address

f3wgfwtrs5p6jrkwfl2mksqa2ivgbgdjjrhjbefy3n7qzvotc3y6sazmp5gfyj7um6jlgdvlbiepzawnc6wxtq

Datacap Allocated

4.20TiB

Signer Address

f1d4gmpqz3execjj2wvrxuuhvbms5mzh7t7yqrviq

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebcnqj64k7ia35fg7aolccewmiiguxwfwjfunwc6gynaxpdl766pa

cryptowhizzard commented 2 years ago

Hello @laurarenpanda

Thank you for requesting new datacap and joining the Filecoin network with the FIL+ / LDN Program.

It seems you spend your datacap to multiple providers where the majority of the providers do not allow public retrieval of the data. This is against the FIL+ / LDN rules. Therefore I (we) cannot grant you further datacap unfortunately.

Please confirm that you are aware that the data stored should be publicly retrievable and that you are aware that it is your responsibility to check this and make this happen.

Do you intend to make a recovery program to re-upload the data to providers with good reputation and do allow retrieval? If so, can you provide us a list of SP's to check?

root@proposals:~# lotus net connect f0118317 f0118317 -> {12D3KooWKdBYgkXGDcAU78435gz435UX7KWKvjziJdazbGoVchiN: [/ip4/62.4.68.25/tcp/26134]} ERROR: failed to parse multiaddr "f0118317": must begin with /

root@proposals:~# lotus net connect f0118330 f0118330 -> {12D3KooWFCNwiZovoPkQyc4pvezdZnjrjPaAcwBTuX7n7BRsw33n: [/ip4/80.11.23.28/tcp/32451]} ERROR: failed to parse multiaddr "f0118330": must begin with /

root@proposals:~# lotus net connect f01228000 ERROR: no PeerID for miner

root@proposals:~# lotus net connect f01228008 ERROR: no PeerID for miner

root@proposals:~# lotus net connect f01228009 ERROR: no PeerID for miner

root@proposals:~# lotus net connect f01228065 ERROR: no PeerID for miner

root@proposals:~# lotus net connect f01228087 ERROR: no PeerID for miner

root@proposals:~# lotus net connect f01228100 ERROR: no PeerID for miner

root@proposals:~#

Schermafbeelding 2022-06-07 om 10 01 03

laurarenpanda commented 2 years ago

Hi @cryptowhizzard. Thanks for pointing out this problem.

When we found storage providers, we ensured they understood FIL+ rules and what they should do to get verified deals. And many of them have participated in the Slingshot with excellent reputation and data onboarding experience. The thing is, to be honest, as a Filecoin client, there are very few things we could do, particularly demanding storage providers to continue to provide public retrieval service.

We'll try our best to talk to them one by one about this problem and figure out the main reason behind it. As we know, there could be factors that influence data retrieval results, including technical, geographical, etc. We believe we could fix it together with these SPs, but it may take time. And we'll consider suspending cooperation until they allow public retrieval.

About publicly retrievability, in the current stage of the Filecoin network, there are still some challenges we need to overcome to improve the large data onboarding process, which means facing more difficulties on the retrieval part. To ensure data availability and follow FIL+ rules, all data of FileDrive Datasets store on IPFS first, which can be accessible to anyone in the network without requiring any special permissions or access requirements.

I hope you and other Notaries can understand what kind of situation we face right now, and please believe our willingness to contribute to the ecosystem. This application is really important for our two open-source ongoing projects behind FileDrive Datasets. Always appreciate understanding and support from Notaries.

cryptowhizzard commented 2 years ago

Hi @laurarenpanda

Thank you for your response. I really appreciate that you take time to respond and address the problem at hand, a problem we need to solve and sort out together.

Given the fact that you need some time to talk to the SP's who are not allowing retrieval i am (personally) willing to park that issue for the time being, to give you time and make a plan to propose a restoration / recover program for this data and sort everything out. If you want to continue in the meantime with storing data then i personally think you need to store that data on nodes who allow retrieval with a good reputation. Is this ok and workable for you? If so, i can propose acceptance for the next tranche of allocation of datacap.

If you need help finding SP's to store the data let us know. I know enough SP's with reputation who can help you out here and give them a link to this Git issue to introduce themselves.

laurarenpanda commented 2 years ago

Hi @cryptowhizzard. Thank you for being so understanding. Your suggestion is very reasonable. And, yes, it is workable for us. Actually, we are working on a project about SPs' reputation system, which could encourage SPs to pay more attention to their service quality (storage/retrieval) and offer them more chances to reach out to clients. We could work with more SPs with excellent reputations shortly. If we need your support to find more excellent SPs, I'll let you know. Thanks again for your help.

cryptowhizzard commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebtqpv2ln5c5gocgluk4j72e2oralafs376znnr6ok3pp6rfxiytg

Address

f3wgfwtrs5p6jrkwfl2mksqa2ivgbgdjjrhjbefy3n7qzvotc3y6sazmp5gfyj7um6jlgdvlbiepzawnc6wxtq

Datacap Allocated

4.20TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebtqpv2ln5c5gocgluk4j72e2oralafs376znnr6ok3pp6rfxiytg

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Request number 8

Multisig Notary address

f01858410

Client address

f3wgfwtrs5p6jrkwfl2mksqa2ivgbgdjjrhjbefy3n7qzvotc3y6sazmp5gfyj7um6jlgdvlbiepzawnc6wxtq

DataCap allocation requested

1B

large-datacap-requests[bot] commented 2 years ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f3wgfwtrs5p6jrkwfl2mksqa2ivgbgdjjrhjbefy3n7qzvotc3y6sazmp5gfyj7um6jlgdvlbiepzawnc6wxtq

Last two approvers

@cryptowhizzard & IreneYoung

Rule to calculate the allocation request amount

800% of weekly dc amount requested

DataCap allocation requested

1B

Total DataCap granted for client so far

4.99PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

1B

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
168142	38	4.20TiB	11.49	193.42GiB

UnionLabs2020 commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedw46sv4dsscv7as4eai2sqp5tsvuouchl6re3oerqqhspv76gfpw

Address

f3wgfwtrs5p6jrkwfl2mksqa2ivgbgdjjrhjbefy3n7qzvotc3y6sazmp5gfyj7um6jlgdvlbiepzawnc6wxtq

Datacap Allocated

1.00B

Signer Address

f17xdri3wunqgld7dm23e4f3eqsntjakwc47xjo6i

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedw46sv4dsscv7as4eai2sqp5tsvuouchl6re3oerqqhspv76gfpw

galen-mcandrew commented 2 years ago

https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/453

Closing this issue since it has pretty much hit the ceiling of 5PiB, and there is an issue in the front-end not picking up the transaction.

filplus-checker commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: FileDrive Labs
Client: f3wgfwtrs5p6jrkwfl2mksqa2ivgbgdjjrhjbefy3n7qzvotc3y6sazmp5gfyj7um6jlgdvlbiepzawnc6wxtq
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Storage provider should not exceed 25% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ 48.34% of total deal sealed by f0522948 are duplicate data.

⚠️ 46.84% of total deal sealed by f0867300 are duplicate data.

⚠️ 44.88% of total deal sealed by f01228008 are duplicate data.

⚠️ 44.91% of total deal sealed by f01228000 are duplicate data.

⚠️ 49.60% of total deal sealed by f01228087 are duplicate data.

⚠️ 49.60% of total deal sealed by f01228089 are duplicate data.

⚠️ 49.49% of total deal sealed by f01228105 are duplicate data.

⚠️ 49.57% of total deal sealed by f01228100 are duplicate data.

⚠️ 48.02% of total deal sealed by f01228009 are duplicate data.

⚠️ 47.95% of total deal sealed by f01228065 are duplicate data.

⚠️ 45.30% of total deal sealed by f023651 are duplicate data.

⚠️ f023651 has unknown IP location.

⚠️ 45.65% of total deal sealed by f01227975 are duplicate data.

⚠️ 50.80% of total deal sealed by f01114589 are duplicate data.

⚠️ f01114589 has unknown IP location.

⚠️ 24.91% of total deal sealed by f01114827 are duplicate data.

⚠️ 46.92% of total deal sealed by f01114587 are duplicate data.

⚠️ f020378 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f0522948`new`	Singapore, Singapore, SG	1.04 PiB	9.81%	551.05 TiB	48.34%
f0867300`new`	Tokyo, Tokyo, JP	1.01 PiB	9.52%	550.44 TiB	46.84%
f01228008`new`	Sydney, New South Wales, AU	1.00 PiB	9.43%	565.18 TiB	44.88%
f01228000`new`	Seoul, Seoul, KR	1.00 PiB	9.43%	564.43 TiB	44.91%
f01228087`new`	London, England, GB	837.89 TiB	7.71%	422.29 TiB	49.60%
f01228089`new`	Frankfurt am Main, Hesse, DE	837.81 TiB	7.71%	422.29 TiB	49.60%
f01228105`new`	Hong Kong, Central and Western, HK	792.78 TiB	7.29%	400.42 TiB	49.49%
f01228100`new`	San Jose, California, US	792.65 TiB	7.29%	399.73 TiB	49.57%
f01228009`new`	Hong Kong, Central and Western, HK	638.45 TiB	5.87%	331.88 TiB	48.02%
f01228065`new`	Singapore, Singapore, SG	637.48 TiB	5.86%	331.82 TiB	47.95%
f023651`new`	Unknown	473.05 TiB	4.35%	258.76 TiB	45.30%
f01227975	Hong Kong, Central and Western, HK	423.91 TiB	3.90%	230.39 TiB	45.65%
f01114589`new`	Unknown	386.15 TiB	3.55%	189.97 TiB	50.80%
f01114827`new`	Tokyo, Tokyo, JP	200.09 TiB	1.84%	150.25 TiB	24.91%
f01114587	Tokyo, Tokyo, JP	199.98 TiB	1.84%	106.15 TiB	46.92%
f0522364`new`	Al Ain City, Abu Dhabi, AE	98.89 TiB	0.91%	98.89 TiB	0.00%
f0522949`new`	Thessaloníki, Central Macedonia, GR	96.39 TiB	0.89%	96.39 TiB	0.00%
f0118330`new`	Hong Kong, Central and Western, HK	73.28 TiB	0.67%	73.28 TiB	0.00%
f0401135`new`	Helsinki, Uusimaa, FI	68.31 TiB	0.63%	68.31 TiB	0.00%
f0134516`new`	Hong Kong, Central and Western, HK	67.68 TiB	0.62%	67.68 TiB	0.00%
f0118317`new`	Frankfurt am Main, Hesse, DE	65.81 TiB	0.61%	65.81 TiB	0.00%
f01163272	Perm, Perm Krai, RU	10.10 TiB	0.09%	8.45 TiB	16.32%
f09848	Rancho Santa Margarita, California, US	3.83 TiB	0.04%	3.40 TiB	11.03%
f01883179	Philadelphia, Pennsylvania, US	2.60 TiB	0.02%	2.56 TiB	1.50%
f02620	Warsaw, Mazovia, PL	1.92 TiB	0.02%	1.88 TiB	2.03%
f020378	Unknown	1.31 TiB	0.01%	1.30 TiB	0.60%
f010088	Everett, Washington, US	1.25 TiB	0.01%	1.25 TiB	0.00%
f01240	Heerhugowaard, North Holland, NL	1.11 TiB	0.01%	1.08 TiB	2.46%
f0773157	Fremont, California, US	1018.00 GiB	0.01%	1018.00 GiB	0.00%
f019551	Birmingham, England, GB	957.00 GiB	0.01%	957.00 GiB	0.00%
f019362	Wuhan, Hubei, CN	920.00 GiB	0.01%	864.00 GiB	6.09%
f022352	Oslo, Oslo, NO	688.00 GiB	0.01%	688.00 GiB	0.00%
f02576	Copenhagen, Capital Region, DK	656.00 GiB	0.01%	656.00 GiB	0.00%
f099608	Riga, Riga, LV	642.00 GiB	0.01%	642.00 GiB	0.00%
f066596	San Diego, California, US	603.00 GiB	0.01%	563.00 GiB	6.63%
f0166896`new`	Jinrongjie, Beijing, CN	224.00 GiB	0.00%	224.00 GiB	0.00%
f097777	Kivertsi, Volyn, UA	114.00 GiB	0.00%	114.00 GiB	0.00%
f01193812	Boardman, Oregon, US	64.00 GiB	0.00%	64.00 GiB	0.00%
f014768	Singapore, Singapore, SG	56.00 GiB	0.00%	56.00 GiB	0.00%
f07998	Wuhan, Hubei, CN	8.00 GiB	0.00%	8.00 GiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

No more than 25% of unique data are stored with less than 4 providers.

✔️ Data replication looks healthy.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
28.65 TiB	33.50 TiB	1	0.31%
201.03 TiB	749.26 TiB	2	6.89%
326.53 TiB	1.75 PiB	3	16.45%
447.63 TiB	3.42 PiB	4	32.19%
298.10 TiB	2.67 PiB	5	25.11%
213.29 TiB	2.02 PiB	6	19.05%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Verifier
f1xd4hex5q44mkddd2n3wr23jmbv7qyzqdcsyetea	Tianjin Zhihui Marine Technology Co., Ltd	99.61 TiB	1,259	LDN # 126
f1ju5oabz45ceog6e7k5omdj56uspv2pzgghiyzdy	SYNC LIVE JAPAN INC.	52.13 TiB	700	LDN v3 multisig
f3qaipvxrz2gxexc7mcvjsxifmscfiw7c7zfhrmq7 6j5ee4hcbvg3gbtpea7wgz72kkjcjmwzhm5uo2onx yocq	Guazi Dynamic	26.03 TiB	368	LDN v3 multisig
f1pkrmygbvweykpjcut36lf7ewgqdfhjklbhvepda	Protocol Labs ( project: Slingshot Evergreen )	14.63 TiB	476	LDN # 293
f1ymfz2mqdrkrdpjmrwh4qaqtuknfpsq3lp3r3auq	Venus team	14.38 TiB	461	LDN v3 multisig
f1aqbilzkvvtggxd4kui5zamwqrdpfzilbtmg5kdi	Nantong Chunna Textile Co., Ltd	5.57 TiB	180	LDN # 165
f3s3dwh67ps4kxui4mziwvus5ue7p6p324aiebrrp e24st5t334ib5gwlxql6auo7ibrccmppjxxavah46 wdqa	FileDrive	5.00 TiB	152	Philipp Banhardt

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: FileDrive Labs
Client: f3wgfwtrs5p6jrkwfl2mksqa2ivgbgdjjrhjbefy3n7qzvotc3y6sazmp5gfyj7um6jlgdvlbiepzawnc6wxtq
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Storage provider should not exceed 30% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.