search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] Kernelogic - NREL Wind Integration National Dataset (1/2) #982

Closed kernelogic closed 1 year ago

kernelogic commented 2 years ago

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Share a brief history of your project and organization.

I have participated every Slingshot phase and is probably the best performing as a "small individual client". 

Even though Slingshot v2 has ended, there are still strong demand from SPs to onboard useful data. This application is to onboard open dataset from AWS.

I will provide a nice web UI to index all files onboarded and provide ways to retrieve.

I have successfully completed a few LDNs on other datasets and I have record to show I have been following the rules of decentralization and have zero self dealing.

https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/60
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/59
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/46
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/297
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/298
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/304

What is the primary source of funding for this project?

Self-funded, BigD exchange.

What other projects/ecosystem stakeholders is this project associated with?

enterprise-sp-wg, BigD exchange.

Use-case details

Describe the data being stored onto Filecoin

NREL Wind Integration National Dataset. 
Released to the public as part of the Department of Energy's Open Energy Data Initiative, the Wind Integration National Dataset (WIND) is an update and expansion of the Eastern Wind Integration Data Set and Western Wind Integration Data Set. It supports the next generation of wind integration studies.

Total size: 952.4 TiB

Where was the data in this dataset sourced from?

AWS Open dataset

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this. 

https://registry.opendata.aws/nrel-pds-wtk/

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

AWS Open dataset. Creative Commons Attribution 3.0 United States License.

What is the expected retrieval frequency for this data?

Multiple times per year.

For how long do you plan to keep this dataset stored on Filecoin?

18 months.

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

All regions.

How will you be distributing your data to storage providers? Is there an offline data transfer process?

I will upload my prepared CAR files to a web server and coordinate with providers to download and propose offline deals.

Maximum 3 copies per SP entity and maximum of 10 copies for every pieceCID.

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

Beside the previous SPs I have worked with, I also utilize bigD exchange to further decentralize the storage

To name a few from the community that I deal with regularly: PIKNIK, Holon, CabrinaHuang, HarryM, BigBear, j1v, XinAn Xu, WillTechMusing.

From BigD exchange: Mog Li, Devin Chen, DSS Nathanial Marsh, Rabinovitch, Vin K, arockpool Tony

How will you be distributing deals across storage providers?

Evenly across all providers I propose to, if they can handle. If a miner is a notary itself, this notary will receive no more than 20% of the total granted datacap.

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

I have all I need to start making deals.
large-datacap-requests[bot] commented 2 years ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!
large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

raghavrmadya commented 2 years ago

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

750TiB

Client address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Multisig Notary address

f01858410

Client address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

DataCap allocation requested

256TiB

Tom-OriginStorage commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecbmtfi4ccq6wogjouie433is3ag7y4rpevj4yxbil2qzliehdwdm

Address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Datacap Allocated

256.00TiB

Signer Address

f1q6bpjlqia6iemqbrdaxr2uehrhpvoju3qh4lpga

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecbmtfi4ccq6wogjouie433is3ag7y4rpevj4yxbil2qzliehdwdm

Destore2023 commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacea3uu72s7v25fgtr6tadvrk6xywkaxjmwzidijkhxrd3ybxxb7tom

Address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Datacap Allocated

256.00TiB

Signer Address

f1yh6q3nmsg7i2sys7f7dexcuajgoweudcqj2chfi

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea3uu72s7v25fgtr6tadvrk6xywkaxjmwzidijkhxrd3ybxxb7tom

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f01858410

Client address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

DataCap allocation requested

512TiB

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Last two approvers

swatchliu & llifezou

Rule to calculate the allocation request amount

10% of total dc amount requested

DataCap allocation requested

512TiB

Total DataCap granted for client so far

256TiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

4.75PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
3274 2 256TiB 86.81 52.28TiB
liyunzhi-666 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedb53y74oye7woftjhymbk2zyecluwdexxgguktaetpbo5hqfmk5s

Address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Datacap Allocated

512.00TiB

Signer Address

f1pszcrsciyixyuxxukkvtazcokexbn54amf7gvoq

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedb53y74oye7woftjhymbk2zyecluwdexxgguktaetpbo5hqfmk5s

flyworker commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacealkckytrguqvcgoabguza6c3vg2erevfo4ijw2k77y7rpxnlhvoc

Address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Datacap Allocated

512.00TiB

Signer Address

f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacealkckytrguqvcgoabguza6c3vg2erevfo4ijw2k77y7rpxnlhvoc

large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!
large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f01858410

Client address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

DataCap allocation requested

1PiB

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Last two approvers

flyworker & liyunzhi-666

Rule to calculate the allocation request amount

20% of total dc amount requested

DataCap allocation requested

1PiB

Total DataCap granted for client so far

768TiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

4.25PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
24637 11 512TiB 16.39 15.40TiB
xiaoyuaiheshui commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebm5whlysrhb7g7dekdeapvabs4vw24ejidghonrwlxssiysvpn5y

Address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Datacap Allocated

1.00PiB

Signer Address

f122qmy25wdtt5mxd77kndiq7z5x2n3iwiuz2wdsa

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebm5whlysrhb7g7dekdeapvabs4vw24ejidghonrwlxssiysvpn5y

newwebgroup commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacec2juzgsltebswcuo2ropdqp5tjijyao2geyetdtvs3ep2ev3hka6

Address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Datacap Allocated

1.00PiB

Signer Address

f1e77zuityhvvw6u2t6tb5qlnsegy2s67qs4lbbbq

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec2juzgsltebswcuo2ropdqp5tjijyao2geyetdtvs3ep2ev3hka6

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f01858410

Client address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

DataCap allocation requested

2PiB

Id

ffbab51c-2c7c-4a0e-b0b7-e2d4e7f86875

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Last two approvers

newwebgroup & jggapp

Rule to calculate the allocation request amount

40% of total dc amount requested

DataCap allocation requested

2PiB

Total DataCap granted for client so far

1.75PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

3.25PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
47382 12 1PiB 18.48 254.35TiB
NiwanDao commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacec5qh5u3rj5isdfsbj56k7fpgb3n3tkinqzquvc2wkizgbehhocz4

Address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Datacap Allocated

2.00PiB

Signer Address

f1a2lia2cwwekeubwo4nppt4v4vebxs2frozarz3q

Id

ffbab51c-2c7c-4a0e-b0b7-e2d4e7f86875

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec5qh5u3rj5isdfsbj56k7fpgb3n3tkinqzquvc2wkizgbehhocz4

newwebgroup commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceamyq5nwvxln6qbhf6daik4gibfmugspst6tsgccyb7xqcj2wbney

Address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Datacap Allocated

2.00PiB

Signer Address

f1e77zuityhvvw6u2t6tb5qlnsegy2s67qs4lbbbq

Id

ffbab51c-2c7c-4a0e-b0b7-e2d4e7f86875

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceamyq5nwvxln6qbhf6daik4gibfmugspst6tsgccyb7xqcj2wbney

filplus-checker commented 1 year ago

DataCap and CID Checker Report[^1]

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

✔️ Storage provider distribution looks healthy.

Provider Location Total Deals Sealed Percentage Unique Data Duplicate Deals
f01928097new Hong Kong, Central and Western, HK 272.47 TiB 13.98% 272.19 TiB 0.10%
f01926635new Hong Kong, Central and Western, HK 192.94 TiB 9.90% 192.94 TiB 0.00%
f01939387 Chengdu, Sichuan, CN 173.53 TiB 8.90% 173.53 TiB 0.00%
f01923786 Hong Kong, Central and Western, HK 130.38 TiB 6.69% 129.41 TiB 0.74%
f01939377 Chengdu, Sichuan, CN 128.27 TiB 6.58% 128.27 TiB 0.00%
f01943663 Hong Kong, Central and Western, HK 125.97 TiB 6.46% 125.97 TiB 0.00%
f01885088 Hong Kong, Central and Western, HK 121.56 TiB 6.24% 121.53 TiB 0.03%
f01923787 Shenzhen, Guangdong, CN 118.88 TiB 6.10% 118.88 TiB 0.00%
f01859603 Shenzhen, Guangdong, CN 116.19 TiB 5.96% 115.22 TiB 0.83%
f01985745 Dallas, Texas, US 112.50 TiB 5.77% 112.50 TiB 0.00%
f01985775 Dallas, Texas, US 112.50 TiB 5.77% 112.50 TiB 0.00%
f033462 Dallas, Texas, US 112.50 TiB 5.77% 112.50 TiB 0.00%
f01933917new Hong Kong, Central and Western, HK 92.66 TiB 4.75% 92.66 TiB 0.00%
f0872282new Shenzhen, Guangdong, CN 85.78 TiB 4.40% 85.28 TiB 0.58%
f01941622 Chengdu, Sichuan, CN 52.93 TiB 2.72% 52.93 TiB 0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

⚠️ 42.38% of deals are for data replicated across less than 4 storage providers.

Unique Data Size Total Deals Made Number of Providers Deal Percentage
246.86 TiB 246.86 TiB 1 12.67%
156.10 TiB 312.42 TiB 2 16.03%
88.92 TiB 266.80 TiB 3 13.69%
34.25 TiB 137.53 TiB 4 7.06%
64.53 TiB 324.38 TiB 5 16.64%
9.50 TiB 57.03 TiB 6 2.93%
63.94 TiB 447.78 TiB 7 22.97%
19.53 TiB 156.25 TiB 8 8.02%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client Application Total Deals Affected Unique CIDs Verifier
f1h74p5qyycefjsoz7qb75lxodopo7acea7nq7e4y Fei Yan - Kernelogic 1.27 PiB 14,823 LDN v3 multisig
f1x7wsqpj6waymzzfqmu4hh32tyc4pbbqnpwy2ucq Glif auto verified 32.00 GiB 1 Jonathan Schwartz

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f01858410

Client address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

DataCap allocation requested

1.25PiB

Id

a81c81e7-b819-4ff7-959b-bfd41d65f5f5

flyworker commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedzjjqxcnnymcfs3ifsld3bvadpuix6xkdeknx64x7bnvuxz2csge

Address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Datacap Allocated

1.25PiB

Signer Address

f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy

Id

a81c81e7-b819-4ff7-959b-bfd41d65f5f5

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedzjjqxcnnymcfs3ifsld3bvadpuix6xkdeknx64x7bnvuxz2csge

NDLABS-Leo commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 37.56% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

NDLABS-Leo commented 1 year ago

image image image The result of the check bot is good, and the connectivity of the nodes has been spot-checked. The connectivity of the nodes is normal, and Kernelogic’s project is compliant as always.

NDLABS-Leo commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacect6ntd5koxoovowaxy6bwu53v4v45rfch5faapzirb3aksyf2gay

Address

f1k6ebid57tjpreo3n7yjkivccx4gue2m4nt2lbkq

Datacap Allocated

1.25PiB

Signer Address

f1yayfsv6whu3rheviucvventj3y6t542xfpb47ei

Id

a81c81e7-b819-4ff7-959b-bfd41d65f5f5

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacect6ntd5koxoovowaxy6bwu53v4v45rfch5faapzirb3aksyf2gay

C00kies77 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 37.87% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

large-datacap-requests[bot] commented 11 months ago

Thanks for your request! :exclamation: We have found some problems in the information provided. We could not find Organization Name field in the information provided We could not find Website \/ Social Media field in the information provided We could not find Total amount of DataCap being requested (between 500 TiB and 5 PiB) field in the information provided We could not find Weekly allocation of DataCap requested (usually between 1-100TiB) field in the information provided We could not find On-chain address for first allocation field in the information provided We could not find Data Type of Application field in the information provided

Please, take a look at the request and edit the body of the issue providing all the required information.