search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] Cabrina-HRRR Open Dataset <7/7> #1146

Closed NiwanDao closed 1 year ago

NiwanDao commented 1 year ago

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Share a brief history of your project and organization.


- I am an active participant in Slingshot and Slingshot Restore. This experience has gained me a lot of knowledge as a data preparer, deal SP, and retrieval client. 
- I have established a relationship with other community members along the way and have successfully sent deals with over 60 SPs worldwide.
- With the surge of requests from other SPs on deal-making and the value of storing humanity’s most important data permanently, I decided to bring HRRR datasets to the network.
- I will track deals and provide retrieval access through https://dstorage.cabrina.xyz/.

What is the primary source of funding for this project?

Mostly self-funded and might be from BigD exchange.

What other projects/ecosystem stakeholders is this project associated with?

No

Use-case details

Describe the data being stored onto Filecoin


> The High-Resolution Rapid Refresh (HRRR) is sourced by Global System Laboratory from National Oceanic && Atmospheric Administration. It is a NOAA real-time 3-km resolution, hourly updated, cloud-resolving, convection-allowing atmospheric model, initialized by 3km grids with 3km radar assimilation. Radar data is assimilated in the HRRR every 15 min over a 1-h period adding further detail to that provided by the hourly data assimilation from the 13km radar-enhanced Rapid Refresh. 
> HRRR dataset stored on AWS is the archive since 2014 with a total size of 2PiB and 38526457 Object.
> I plan to send 10x copies, each for 2PiB raw data. Since there is a conversion rate between the raw data size and Datacap consumption size, each copy would require around 3.5PiB Datacap, with a total amount of  35PiB Datacap.

Where was the data in this dataset sourced from?

AWS: https://registry.opendata.aws/noaa-hrrr-pds/

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this. 

Link a video on how HRRR is critical to forecasting weather. 
https://www.youtube.com/watch?v=tIPHkPeW7CA

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

Confirmed

What is the expected retrieval frequency for this data?

Not often

For how long do you plan to keep this dataset stored on Filecoin?

> 360 days.

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

Welcome storage providers from all countries

How will you be distributing your data to storage providers? Is there an offline data transfer process?

I am open to both options: offline and online. 
For storage providers in China, offline delivery might be a better choice from a speed perspective. 
For others, distributing data through HTTP might be more realistic.

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

I will consider SPs I have worked with before and am also willing to partner with other SPs that demonstrate the ability to run node safely,  handle real deals with a consistent sealing rate, and supports retrieval.

How will you be distributing deals across storage providers?

Each SP can store no more than 1 copy of data, which means no single SP weighs more than 10% of all 35PiB I proposed.

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

yes, I am ready to go
newwebgroup commented 1 year ago
  1. CID Checker's results are more compliant than before
  2. Tested CID retrieval, normal lotus client retrieve --provider f01988794 Qmcjvxo9s8VKcVN2t2FAwPtuN51e7T8j19nRyAghbGYa2a LDN-1146-1 image

3.After reviewing the Github history, the LDN was verified and supported by several notaries, so it is willing to support again.

newwebgroup commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacea5bburr7sjba3mfil2z2rifwrh5ux6ruqpxn7olawn6xrehgdmwi

Address

f1nyllpc6mxc4sdqfgcny75e2lv7onr4qw5p7h44y

Datacap Allocated

1.25PiB

Signer Address

f1e77zuityhvvw6u2t6tb5qlnsegy2s67qs4lbbbq

Id

f5b19416-615c-4086-bd37-8cc09db8fae9

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea5bburr7sjba3mfil2z2rifwrh5ux6ruqpxn7olawn6xrehgdmwi

woshidama323 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

woshidama323 commented 1 year ago

It looks well at the moment, Will support this application

flyworker commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacednxdhrajkcpp4y2i5ivfb6lvqniyd4ngpwoxasrc3rkzzmp2ef66

Address

f1nyllpc6mxc4sdqfgcny75e2lv7onr4qw5p7h44y

Datacap Allocated

1.25PiB

Signer Address

f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy

Id

f5b19416-615c-4086-bd37-8cc09db8fae9

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacednxdhrajkcpp4y2i5ivfb6lvqniyd4ngpwoxasrc3rkzzmp2ef66

NiwanDao commented 1 year ago

@Kevin-FF-USA @raghavrmadya Notaries proposed and approved the LDN, but the client address has not received any new DataCap. Please help here.

截屏2023-03-17 下午8 19 10
stcloudlisa commented 1 year ago
WechatIMG239
stcloudlisa commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecautqi6myifwciubdpmn6q3qzk3e376aofvrewne55bgz35mlr4k

Address

f1nyllpc6mxc4sdqfgcny75e2lv7onr4qw5p7h44y

Datacap Allocated

1.25PiB

Signer Address

f1jvvltduw35u6inn5tr4nfualyd42bh3vjtylgci

Id

f5b19416-615c-4086-bd37-8cc09db8fae9

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecautqi6myifwciubdpmn6q3qzk3e376aofvrewne55bgz35mlr4k

NiwanDao commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

⚠️ 1 storage providers have unknown IP location - f02031063

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

NiwanDao commented 1 year ago

WIP

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

NiwanDao commented 1 year ago

checker:manualTrigger

NiwanDao commented 1 year ago

WIP

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!