filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
109 stars 62 forks source link

[DataCap Application] Fly Brain Dataset #1013

Closed Megan008 closed 1 year ago

Megan008 commented 2 years ago

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Share a brief history of your project and organization.

I have participated in some projects and hackathon. I have experience on it.

What is the primary source of funding for this project?

Personal income.

What other projects/ecosystem stakeholders is this project associated with?

No.

Use-case details

Describe the data being stored onto Filecoin

It consists of fluorescence images of Drosophila melanogaster driver lines, aligned to standard templates, and stored in formats suitable for rapid searching in the cloud.

Where was the data in this dataset sourced from?

This data set is made available by Janelia's FlyLight project.

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this.

https://registry.opendata.aws/janelia-flylight/

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

Yes, it's a public dataset

What is the expected retrieval frequency for this data?

Multiple times.

For how long do you plan to keep this dataset stored on Filecoin?

2 years.

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

North america; Korea; China.

How will you be distributing your data to storage providers? Is there an offline data transfer process?

75% data will be distributed by offline data transfer. Other data will use online transfer for distributing with storage providers who close to me.

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

I would let 1 sp who used to cooperate with me for this deal. Now I'm chatting with other sps. f023495, f0508988

How will you be distributing deals across storage providers?

I have communicated with 4 sp. In first time, I will divide 1/4 data to each sp. If I find out more sp, I will decrease the percentage of deals to them --- for decentralized storage.

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

Yes.
large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

raghavrmadya commented 2 years ago
Screen Shot 2022-10-07 at 8 26 39 AM

The data set is already stored on slingshot and the size does not justify 5 PiBs. Closing this application now

Megan008 commented 2 years ago

@raghavrmadya It is a public dataset so that anyone can store it not only for slingshot. The actual size of this dataset is close to 300 TiB, and you can know the size of it via the following code--- aws s3 ls s3://janelia-flylight-imagery/ --no-sign-request --recursive --summarize

342762

The reason I apply for 5 PiB is that I want to store about 10 copies at least and prepare for possible error of storing in the future. Now I changed my total request into 3PiB. Can you help reopen my issue and let me prepare my application again? Thank you!

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

raghavrmadya commented 2 years ago

Datacap Request Trigger

Total DataCap requested

3PiB

Expected weekly DataCap usage rate

100TiB

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Multisig Notary address

f01858410

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

DataCap allocation requested

50TiB

raghavrmadya commented 2 years ago

Can you also share how your data preparation is different than the existing data set?

Megan008 commented 2 years ago

I have enough bandwidth and hard-disks for downloading data.

MetaWaveInfo commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceb4nfhtjp4yy7x7727zbok7byj37w622iw3smzecxbziu4jhcei64

Address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Datacap Allocated

50.00TiB

Signer Address

f1ktlkcxnmzxcdaoqfsunrg3vocfbmgv4n3mrn74a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceb4nfhtjp4yy7x7727zbok7byj37w622iw3smzecxbziu4jhcei64

UnionLabs2020 commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceav5s7orw7urk6ajd5zvisud75u7sfw6fhuyb5ddrsvxb23d36gri

Address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Datacap Allocated

50.00TiB

Signer Address

f17xdri3wunqgld7dm23e4f3eqsntjakwc47xjo6i

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceav5s7orw7urk6ajd5zvisud75u7sfw6fhuyb5ddrsvxb23d36gri

BDE-io commented 2 years ago

@Megan008 Hi! Great to see you have gotten approval for DataCap and advancing the mission of preserving humanity’s most important information. If you are looking for storage providers to store these data or have any questions, please visit #bigdata-exchange on Filecoin Slack or reply here.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f01858410

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

DataCap allocation requested

100TiB

Id

c1363cb1-c52b-4911-a647-0a85fc5a7881

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Last two approvers

UnionLabs2020 & MetaWaveInfo

Rule to calculate the allocation request amount

100% of weekly dc amount requested

DataCap allocation requested

100TiB

Total DataCap granted for client so far

50TiB

Datacap to be granted to reach the total amount requested by the client (3PiB)

2.95PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
0 0 50TiB 0 7.53TiB
Destore2023 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceatan5gyzawhvzcgyl67yzgmxwvrkp3dk5gt3x7sdyut3vykcb6p6

Address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Datacap Allocated

100.00TiB

Signer Address

f1yh6q3nmsg7i2sys7f7dexcuajgoweudcqj2chfi

Id

c1363cb1-c52b-4911-a647-0a85fc5a7881

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceatan5gyzawhvzcgyl67yzgmxwvrkp3dk5gt3x7sdyut3vykcb6p6

Fenbushi-Filecoin commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecyuu4okgohw6foraqf2g464w4xo4grmcs44puscsxh4qa64d6x64

Address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Datacap Allocated

100.00TiB

Signer Address

f1yqydpmqb5en262jpottko2kd65msajax7fi4rmq

Id

c1363cb1-c52b-4911-a647-0a85fc5a7881

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecyuu4okgohw6foraqf2g464w4xo4grmcs44puscsxh4qa64d6x64

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f01858410

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

DataCap allocation requested

200TiB

Id

7784679a-4452-4c81-962c-e455f83ff774

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Last two approvers

Fenbushi-Filecoin & swatchliu

Rule to calculate the allocation request amount

200% of weekly dc amount requested

DataCap allocation requested

200TiB

Total DataCap granted for client so far

150TiB

Datacap to be granted to reach the total amount requested by the client (3PiB)

2.85PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
879 1 100TiB 100.00 19.46TiB
MetaWaveInfo commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacea6ixuehbcds5wcj2b2izc5zbbyl5jyjok7cxbdgcxs2ilvgsbksu

Address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Datacap Allocated

200.00TiB

Signer Address

f1ktlkcxnmzxcdaoqfsunrg3vocfbmgv4n3mrn74a

Id

7784679a-4452-4c81-962c-e455f83ff774

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea6ixuehbcds5wcj2b2izc5zbbyl5jyjok7cxbdgcxs2ilvgsbksu

UnionLabs2020 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecwcirkmvivbqeo3bfpdndizjtiwsud3zedc6ynktor62hyaxi7ok

Address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Datacap Allocated

200.00TiB

Signer Address

f17xdri3wunqgld7dm23e4f3eqsntjakwc47xjo6i

Id

7784679a-4452-4c81-962c-e455f83ff774

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecwcirkmvivbqeo3bfpdndizjtiwsud3zedc6ynktor62hyaxi7ok

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f01858410

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

DataCap allocation requested

400TiB

Id

367492a5-0125-4f85-bc8c-37c8ace9bb41

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Last two approvers

UnionLabs2020 & MetaWaveInfo

Rule to calculate the allocation request amount

400% of weekly dc amount requested

DataCap allocation requested

400TiB

Total DataCap granted for client so far

150TiB

Datacap to be granted to reach the total amount requested by the client (3PiB)

2.85PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
879 1 200TiB 100.00 43.12TiB
Destore2023 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceckydimni7np4c6snbn2gqagcwznbkxzwbwbdabkaedvp6523nkfo

Address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Datacap Allocated

400.00TiB

Signer Address

f1yh6q3nmsg7i2sys7f7dexcuajgoweudcqj2chfi

Id

367492a5-0125-4f85-bc8c-37c8ace9bb41

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceckydimni7np4c6snbn2gqagcwznbkxzwbwbdabkaedvp6523nkfo

stcloudlisa commented 1 year ago

WechatIMG355

stcloudlisa commented 1 year ago

1669876790719

stcloudlisa commented 1 year ago

It seems that there is an error in the browser, but I checked the customer's situation, and the customer is in compliance with the rules

stcloudlisa commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedn6npsts54zfqhrkuw2xsx6n52f6jhhkjq2tkkd4zmgm2u2cz5iu

Address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Datacap Allocated

400.00TiB

Signer Address

f1jvvltduw35u6inn5tr4nfualyd42bh3vjtylgci

Id

367492a5-0125-4f85-bc8c-37c8ace9bb41

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedn6npsts54zfqhrkuw2xsx6n52f6jhhkjq2tkkd4zmgm2u2cz5iu

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f01858410

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

DataCap allocation requested

800TiB

Id

ea2cab80-b2d5-434b-a5f9-1a96b698a971

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Last two approvers

1LISA2 & swatchliu

Rule to calculate the allocation request amount

800% of weekly dc amount requested

DataCap allocation requested

800TiB

Total DataCap granted for client so far

750TiB

Datacap to be granted to reach the total amount requested by the client (3PiB)

2.26PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
18603 6 400TiB 45.70 99TiB
NDLABS-Leo commented 1 year ago

image

UnionLabs2020 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceduuctcvd2mq643qvrutnxeywwocrttwv663ov4q722zjl26mbsrg

Address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Datacap Allocated

800.00TiB

Signer Address

f17xdri3wunqgld7dm23e4f3eqsntjakwc47xjo6i

Id

ea2cab80-b2d5-434b-a5f9-1a96b698a971

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceduuctcvd2mq643qvrutnxeywwocrttwv663ov4q722zjl26mbsrg

MetaWaveInfo commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebziecq6632htqw4nkewbtnkw7of5bmmmhx7d3cn77rlyf7q2tlhg

Address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Datacap Allocated

800.00TiB

Signer Address

f1ktlkcxnmzxcdaoqfsunrg3vocfbmgv4n3mrn74a

Id

ea2cab80-b2d5-434b-a5f9-1a96b698a971

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebziecq6632htqw4nkewbtnkw7of5bmmmhx7d3cn77rlyf7q2tlhg

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 6

Multisig Notary address

f01858410

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

DataCap allocation requested

800TiB

Id

c3ff5c1b-45a9-43c0-9f9a-b5e385f31218

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1cf2pp6bgu6vvxfuau6bsmiibhrc7v3gvjsyseay

Last two approvers

MetaWaveInfo & UnionLabs2020

Rule to calculate the allocation request amount

800% of weekly dc amount requested

DataCap allocation requested

800TiB

Total DataCap granted for client so far

750TiB

Datacap to be granted to reach the total amount requested by the client (3PiB)

2.26PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
18957 6 800TiB 44.95 90.40TiB
filplus-checker commented 1 year ago

DataCap and CID Checker Report[^1]

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

⚠️ f01969789 has sealed 46.65% of total datacap.

⚠️ 79.86% of total deal sealed by f01969789 are duplicate data.

⚠️ 60.84% of total deal sealed by f01344987 are duplicate data.

⚠️ 89.94% of total deal sealed by f0136399 are duplicate data.

⚠️ 46.19% of total deal sealed by f01841131 are duplicate data.

⚠️ 89.91% of total deal sealed by f0229199 are duplicate data.

⚠️ 89.75% of total deal sealed by f0681068 are duplicate data.

⚠️ All storage providers are located in the same region.

Provider Location Total Deals Sealed Percentage Unique Data Duplicate Deals
f01969789 Hong Kong, Central and Western, HK 304.91 TiB 46.65% 61.41 TiB 79.86%
f01344987 Hong Kong, Central and Western, HK 119.31 TiB 18.25% 46.72 TiB 60.84%
f0136399 Hong Kong, Central and Western, HK 115.50 TiB 17.67% 11.63 TiB 89.94%
f01841131 Hong Kong, Central and Western, HK 59.06 TiB 9.04% 31.78 TiB 46.19%
f0229199 Hong Kong, Central and Western, HK 34.69 TiB 5.31% 3.50 TiB 89.91%
f0681068 Hong Kong, Central and Western, HK 20.13 TiB 3.08% 2.06 TiB 89.75%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

⚠️ 100.00% of deals are for data replicated across less than 4 storage providers.

Unique Data Size Total Deals Made Number of Providers Deal Percentage
132.16 TiB 571.09 TiB 1 87.38%
12.47 TiB 82.50 TiB 2 12.62%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client Application Total Deals Affected Unique CIDs Verifier
f1znd7uramgqquelcgefo5xgozopd3nga73hhywpy BotSmart 82.94 TiB 343 LDN v3 multisig
f1tgii4xgcp6um4n2w7eqaw6bwvnvyii2u7cjgzka Meson.Network 54.34 TiB 157 LDN v3 multisig
f127l5waa4pw6k3sm4e7wxevc4fe5uz7dyt4mxa3i Pangod 52.69 TiB 309 LDN v3 multisig
f1quofpadzcqlonmz7v3mv7dfqvm5hdztucdsjqsy HENAN 863 SOFTWARE CO., LTD 47.63 TiB 141 LDN v3 multisig
f1vbe7ze5yknpgu3orinv4zu4rkofsyqfsmpejpui HELIOS 46.56 TiB 151 LDN v3 multisig
f1wr6rwwqckh6um2hynaym2t4mniev5b675kbf5ni Ctrip Global Shopping 46.47 TiB 230 LDN v3 multisig
f1ugo3abkmmb4pb2atxz5oqqgwsd27b4p6k52f2va Yuepass Technology Company Limited 42.09 TiB 285 LDN v3 multisig
f1bb2z36lpq3pnwiiowiraagpzqnpow4bonjacx7a Hola Space 23.41 TiB 75 LDN v3 multisig
f1ibuglt2lzlyf7vnmtzmuykcfgl2pn2fxym5dibi Appstest 21.88 TiB 175 LDN v3 multisig
f1csetl7nor3qie2cehx7axf2ai3nedmowj53xwsa NOAA GOES---Piero 18.81 TiB 154 LDN v3 multisig
f1vykg3elgzoa3lpzf5xo6rbcghzr5ifat757mucq Hailiang Mingyou Online 10.81 TiB 71 LDN # 51
f1a2rdwwor3kq6mv7nveuxhux7rxtj6iyjs7hfswa Worldkan 6.59 TiB 42 LDN v3 multisig
f1rkmhotssjif6ucrosls7oewjz6pr2v2eygfjyui Weipaitang 5.66 TiB 34 LDN v3 multisig
f1fq6abg47ifgeee2z7q2rps3tvknoo2ztcoqy7ai DaYe Art Tuition Class (DKArt) 864.00 GiB 5 LDN v3 multisig
f3u3unadf654vezf62cd4jo6r7h6qpkx26g5amcdc
3oe6rmpmk2nfosfd2kjkdhj4ndvr626gsm7fhpmt7
gg2q
Runtu Information Technology 608.00 GiB 4 Steven Li

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

BDEio commented 1 year ago

@Megan008 Hi! Great to see that you have gotten approval for DataCap! BDE is a verified deals auction house helping you to get paid storing your valuable data with reliable storage providers. If you need any help, please get in touch.

aggregation-and-compliance-bot[bot] commented 11 months ago
Client f01925065 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 7 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Cid Checker score > 25% The client has a CID checker score of 8%. This should be greater than 25%. To find out more about CID checker score please look at this issue: https://github.com/filecoin-project/notary-governance/issues/986
large-datacap-requests[bot] commented 10 months ago

Thanks for your request! :exclamation: We have found some problems in the information provided. We could not find Website \/ Social Media field in the information provided We could not find Total amount of DataCap being requested (between 500 TiB and 5 PiB) field in the information provided We could not find Weekly allocation of DataCap requested (usually between 1-100TiB) field in the information provided We could not find On-chain address for first allocation field in the information provided We could not find Data Type of Application field in the information provided

Please, take a look at the request and edit the body of the issue providing all the required information.
large-datacap-requests[bot] commented 8 months ago

RootKeyHolders have approved multisig account. You can now request first datacap release

large-datacap-requests[bot] commented 4 months ago

RootKeyHolders have approved multisig account. You can now request first datacap release