search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] <DataMe> - <World Economic Data> #2026

Closed QodeNu closed 1 year ago

QodeNu commented 1 year ago

Data Owner Name

DataMe

What is your role related to the dataset

Data Preparer

Data Owner Country/Region

Hong Kong

Data Owner Industry

IT & Technology Services

Website

http://www.econostatistics.co.za/

Social Media

n/a

Total amount of DataCap being requested

10PiB

Expected size of single dataset (one copy)

2P

Number of replicas to store

10

Weekly allocation of DataCap requested

1PiB

On-chain address for first allocation

f1cgqp6ivn2expwksf4hmbv6qp2v6gi4r2qh6umzy

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Identifier

No response

Share a brief history of your project and organization

DATAME was established in Hong Kong in 2021. We focus on the decentralized storage track. In the past, we focused on cc packaging. Currently, we have in total of 50P storage power on Filecoin. Now we are ready to upgrade the software and hardware from cc to dc. I am a pure Filecoin machine and computing power investor, without official website and promotional. Thank you for your undestanding.
Economic data is crucial in the process of human development and affects the development of countries and individuals. We plan to upload economic data that is critical to human development to the Filecoin network. We have collected 30 large-scale economics-related datasets, in a total of 20P storage capacity, with 10 backups.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

Economic data is crucial in the process of human development and affects the development of countries and individuals. We plan to upload economic data that is critical to human development to the Filecoin network. We have collected 30 large-scale economics-related datasets, in a total of 20P storage capacity, with 10 backups.
Dataset including but not all: 
Asian Productivity Organization (APO) 
ASEAN Stats 
American Economic Association (AEA)
Asian KLEMS 
Harvard Atlas of Economic Complexity 
BIS Financial Database
Barro-Lee Education Attainment - Barro-Lee Educational Attainment Data from 1950 to 2010
CEPII Database 
EUKLEMS - EU KLEMS is an industry level, growth and productivity research project. EU KLEMS
Economic Freedom of the World Data
Latin America KLEMS 
Long-Term Productivity Database 
Maddison Project Database
National Transfer Accounts 
OpenCorporates Database of Companies in the World
Our World in Data [Meta]
Penn World Table - PWT version 10.0

Where was the data currently stored in this dataset sourced from

My Own Storage Infra

If you answered "Other" in the previous question, enter the details here

No response

How do you plan to prepare the dataset

IPFS, lotus, singularity

If you answered "other/custom tool" in the previous question, enter the details here

No response

Please share a sample of the data

https://www.aeaweb.org/resources/data
https://www.apo-tokyo.org/
https://data.aseanstats.org/
https://dataverse.harvard.edu/dataverse/atlas
http://www.historicalstatistics.org/
http://www.econostatistics.co.za/
https://www.upcdatabase.com/
http://www.jedh.org/
https://www.rug.nl/ggdc/valuechain/wiod/
https://www.fraserinstitute.org/economic-freedom/dataset?geozone=world&page=dataset&min-year=2&max-year=0&filter=0

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

2 to 3 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, North America, South America, Europe

How will you be distributing your data to storage providers

Cloud storage (i.e. S3), HTTP or FTP server, IPFS, Shipping hard drives, Lotus built-in data transfer

How do you plan to choose storage providers

Slack, Partners

If you answered "Others" in the previous question, what is the tool or platform you plan to use

No response

If you already have a list of storage providers to work with, fill out their names and provider IDs below

f02199393/f02192496/f02095766/f01971431/f02115125/f02185816 so far

How do you plan to make deals to your storage providers

Boost client, Lotus client, Droplet client, Singularity

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

Sunnyiscoming commented 1 year ago

You are applying for a total amount of 10 PiB, but the single dataset is 2PiB and the number of replicas is 10. (2 PiB x 10 copies) = 20 PiB Why do you apply for 10 PiB and not 20 PiB?

Could you send your business license to filplus-app-review@fil.org in order to confirm your identity? Email name should includes the issue id #2026?

QodeNu commented 1 year ago

@Sunnyiscoming Cause I wanna go through step by step. I'm not sure whether the process is smooth or not, even though I have much more data. Looking forward your support. Thx!

yup, sure, I have sent the business licences through my email.

Sunnyiscoming commented 1 year ago

Datacap Request Trigger

Total DataCap requested

10PiB

Expected weekly DataCap usage rate

1PiB

Client address

f1cgqp6ivn2expwksf4hmbv6qp2v6gi4r2qh6umzy

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1cgqp6ivn2expwksf4hmbv6qp2v6gi4r2qh6umzy

DataCap allocation requested

512TiB

Id

b8fce6c5-8fc8-4ce9-af06-3bd54ef34d85

kernelogic commented 1 year ago

In support for public data

laurarenpanda commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecpba32guma6uqajq5sqczrviorcv7l5ye5cnsjs76y34x4dnvkic

Address

f1cgqp6ivn2expwksf4hmbv6qp2v6gi4r2qh6umzy

Datacap Allocated

512.00TiB

Signer Address

f1bp3tzp536edm7dodldceekzbsx7zcy7hdfg6uzq

Id

b8fce6c5-8fc8-4ce9-af06-3bd54ef34d85

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecpba32guma6uqajq5sqczrviorcv7l5ye5cnsjs76y34x4dnvkic

kernelogic commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebklrbhq46iqguqwcikuovg7gtdkhj5agm6qvhhhp66q6l5xlt34y

Address

f1cgqp6ivn2expwksf4hmbv6qp2v6gi4r2qh6umzy

Datacap Allocated

512.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

b8fce6c5-8fc8-4ce9-af06-3bd54ef34d85

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebklrbhq46iqguqwcikuovg7gtdkhj5agm6qvhhhp66q6l5xlt34y

Sunnyiscoming commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

spaceT9 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

newwebgroup commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

zcfil commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 2 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1cgqp6ivn2expwksf4hmbv6qp2v6gi4r2qh6umzy

DataCap allocation requested

512TiB

Id

2d18bdcd-0bf0-4b60-af6b-d9c2500c7baf

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1cgqp6ivn2expwksf4hmbv6qp2v6gi4r2qh6umzy

Rule to calculate the allocation request amount

100% weekly > 0.5PiB, requesting 0.5PiB

DataCap allocation requested

512TiB

Total DataCap granted for client so far

512TiB

Datacap to be granted to reach the total amount requested by the client (10PiB)

9.5PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
10588 4 512TiB 27.61 121.34TiB
filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

nj-steve commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

nj-steve commented 1 year ago

Hello @QodeNu Have you contacted the SPs? Are they all in favor of retrieval？The report shows that they all do not support retrieval.

QodeNu commented 1 year ago

Hello, @nj-steve Thank you for your attention. After I discovered this problem yesterday, I contacted these sps, and I informed them that they all should support retrieval, which is the newest rule in Fil+. It has been confirmed today that the node is retrievable. I hope you can support us and keep an eye on the latest updates, thank you.

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

ipollo00 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

ipollo00 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

sxxfuture-official commented 1 year ago

@QodeNu According to the CID check report, so far all the data has only been sealed on one SP, can you explain this situation? In addition, what plans do you have to solve this problem in the future?

QodeNu commented 1 year ago

Hi @sxxfuture-official The copies of data have been sent to various SPs through online and offline transfers at different speeds. It will definitely be shown across SPs gradually. Thx.

sxxfuture-official commented 1 year ago

Gotta fix things soon, I'll support this round, but I'll keep an eye on it.

sxxfuture-official commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceczv3xcjc2rin3by25khkydpys3uzz7jpnzj46usbsg4ygjpopdsg

Address

f1cgqp6ivn2expwksf4hmbv6qp2v6gi4r2qh6umzy

Datacap Allocated

512.00TiB

Signer Address

f1foiomqlmoshpuxm6aie4xysffqezkjnokgwcecq

Id

2d18bdcd-0bf0-4b60-af6b-d9c2500c7baf

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceczv3xcjc2rin3by25khkydpys3uzz7jpnzj46usbsg4ygjpopdsg

DirectionTechnology commented 1 year ago

From the recent CID report, it is evident that there has been a significant improvement in Graphsync retrieval. I would like to know what plans you have in improving HTTP/Bitswap retrieval? @QodeNu

QodeNu commented 1 year ago

It depends. As for now, we may using Lotus combining with the lates HTTP code updating in Boost Market. Will keep in touch with latest news in community.

DirectionTechnology commented 1 year ago

Hope to see tangible results in these regards in the next round.

DirectionTechnology commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebefhylz73bmecyktmmhgxo6x4ixxkw3awbelw4lfpghkkmmb2r4g

Address

f1cgqp6ivn2expwksf4hmbv6qp2v6gi4r2qh6umzy

Datacap Allocated

512.00TiB

Signer Address

f1inkdoatsbfumdvpctxbgcatscewr3rus5pxmsgi

Id

2d18bdcd-0bf0-4b60-af6b-d9c2500c7baf

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebefhylz73bmecyktmmhgxo6x4ixxkw3awbelw4lfpghkkmmb2r4g

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1cgqp6ivn2expwksf4hmbv6qp2v6gi4r2qh6umzy

DataCap allocation requested

1PiB

Id

861f77a9-5dbe-4d19-a717-463a26bb3dea

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1cgqp6ivn2expwksf4hmbv6qp2v6gi4r2qh6umzy

Rule to calculate the allocation request amount

200% weekly > 1PiB, requesting 1PiB

DataCap allocation requested

1PiB

Total DataCap granted for client so far

465661.3YiB

Datacap to be granted to reach the total amount requested by the client (10PiB)

465661.3YiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
22132 10 512TiB 22.91 126.59TiB
woshidama323 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 97.28% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

Normalnoise commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 97.28% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

woshidama323 commented 1 year ago

what's your plan for fixing this warning?

QodeNu commented 1 year ago

Hello, @woshidama323 as you can see, data backup has begun to be uploaded by various SPs. Because this application form is in the early stage, SPs have different speeds to process data. In the end, it will be stored in accordance with the description of the application form. Please support, thank you.