search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] NOAA-HRRR #2116

Closed FroghubMan closed 7 months ago

FroghubMan commented 1 year ago

Data Owner Name

National Oceanic and Atmospheric Administration

What is your role related to the dataset

Data Preparer

Data Owner Country/Region

United States

Data Owner Industry

Environment

Website

https://registry.opendata.aws/noaa-hrrr-pds

Social Media

Website: https://www.noaa.gov/
Twitter: https://twitter.com/NOAA

Total amount of DataCap being requested

15PiB

Expected size of single dataset (one copy)

2.1

Number of replicas to store

7

Weekly allocation of DataCap requested

1PiB

On-chain address for first allocation

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Identifier

No response

Share a brief history of your project and organization

FrogHub files is a platform that will provide users with free storage, allowing users to enjoy free storage at the same time. Facilitating the Filecoin network to store more and more valuable public data sets.

About the FrogHub data set

FrogHub files is a user-oriented cloud storage platform. Any user with data storage needs can use the platform tools free of charge.

The Filecoin Network provides reliable, secure, and affordable decentralized storage, and FrogHub files wants to make it easy for users to store their data in the filecoin network.

Attracting traditional cloud and object-based storage users to the Filecoin network and benefiting from it is a challenge. FrogHub, as developers in the Felicoin ecosystem, needs to face this challenge together. We've been building useful tools to make it easier for users to store data on the Filecoin network.

Froghub files integrates a set of tools, including storage bucket, custom gateway, data encryption service KMS, static web deployment (under development), open API, to provide storage services compatible with cloud storage and object storage, providing better user experience and attracting more users.
FrogHub has always defined itself as a tool developer and infrastructure builder in the Filecoin ecosystem. In 2019, we started to focus on technical solutions and development based on the IPFS protocol and Filecoin network. We have been working hard to become a qualified builder in the filecoin ecosystem.

Our team is a very pure development team, more than 90% of which are developers, more than half of whom have more than 5 years of development experience in communication, Internet, blockchain and other industries. We hope that we can gain users' recognition by exporting useful tools and platforms.

In order to contribute to the filecoin community, we have developed the open source sector repair tool Filecoin-Sealer-Recover and the nft free authoring platform NFT-Creator.
In addition, we plan to provide a sector browser for the community in 2023 and build the liquidity pledge platform STFIL on FVM.

See the links below for details.
- github: https://github.com/orgs/froghub-io
- twitter: https://twitter.com/froghub_io

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

357-resule 
2020-08-25 17:22:47  398.4 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf13.grib2
2020-08-25 17:22:52  396.0 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf14.grib2
2020-08-25 17:22:52  394.6 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf15.grib2
2020-08-25 17:22:52  390.2 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf16.grib2
2020-08-25 17:23:08  387.1 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf17.grib2
2020-08-25 17:23:05  384.8 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf18.grib2
2021-09-28 03:48:22   31.6 KiB index.html

Total Objects: 43282174
   Total Size: 2.1 PiB

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

No response

If you are a data preparer. What is your location (City and Country)

No response

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

No response

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

No response

Please share a sample of the data

357-resule 
2020-08-25 17:22:47  398.4 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf13.grib2
2020-08-25 17:22:52  396.0 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf14.grib2
2020-08-25 17:22:52  394.6 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf15.grib2
2020-08-25 17:22:52  390.2 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf16.grib2
2020-08-25 17:23:08  387.1 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf17.grib2
2020-08-25 17:23:05  384.8 MiB hrrr_v2.20160823/conus/hrrr.t09z.wrfprsf18.grib2
2021-09-28 03:48:22   31.6 KiB index.html

Total Objects: 43282174
   Total Size: 2.1 PiB

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

2 to 3 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, North America, Europe

How will you be distributing your data to storage providers

Cloud storage (i.e. S3), Shipping hard drives

How do you plan to choose storage providers

Slack, Big Data Exchange, Partners

If you answered "Others" in the previous question, what is the tool or platform you plan to use

No response

If you already have a list of storage providers to work with, fill out their names and provider IDs below

No response

How do you plan to make deals to your storage providers

No response

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

Sunnyiscoming commented 1 year ago
FroghubMan commented 1 year ago
  1. I am a data preparer, and the SP I am working with has enough FIL as storage collateral.
  2. We want to store valuable data. The data for this application is something we have not stored before.
  3. The following are the SPs we have cooperated with.
MinerID City Continent
f01811024 HongKong CN
f0827006 Tokyo Japan
f02227726 HongKong CN
f02195153 Tokyo Japan
f02098006 Krabi Thailand
f02223012 Hebei CN
f02235154 HongKong CN
f02085722 Jiangmen CN
  1. I have sent the confirmation email.
ghost commented 1 year ago

Left a comment on 2119

FroghubMan commented 1 year ago

Hi @Filplus-govteam @Sunnyiscoming ,I have submitted the information in #2119, please review it for me. Thank you very much.

FroghubMan commented 1 year ago

Hi @Sunnyiscoming @Filplus-govteam , I have completed the Fil+ registration form once again. Can you review it for me? Thank you so much.

Sunnyiscoming commented 1 year ago

Datacap Request Trigger

Total DataCap requested

15PiB

Expected weekly DataCap usage rate

1PiB

Client address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

DataCap allocation requested

512TiB

Id

750561f3-d6c5-4957-bea9-11c3eb33c768

MatrixStorage commented 1 year ago

I am willing to support

MatrixStorage commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecqdoqdvubibbn4ngp4bulytcddrzkgiydh2iv5q7m5yckgdcm3la

Address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

Datacap Allocated

512.00TiB

Signer Address

f1tbxqwjxfyv7swsdin4einirlsfquv3vnmlapley

Id

750561f3-d6c5-4957-bea9-11c3eb33c768

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecqdoqdvubibbn4ngp4bulytcddrzkgiydh2iv5q7m5yckgdcm3la

FroghubMan commented 1 year ago

Thank you

FroghubMan commented 1 year ago

In progress

stcloudlisa commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebzvkt3rwqnwheoymemd4vfwazndtfag5bptlbd4bxm7pvbyso2uc

Address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

Datacap Allocated

512.00TiB

Signer Address

f1jvvltduw35u6inn5tr4nfualyd42bh3vjtylgci

Id

750561f3-d6c5-4957-bea9-11c3eb33c768

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebzvkt3rwqnwheoymemd4vfwazndtfag5bptlbd4bxm7pvbyso2uc

FroghubMan commented 1 year ago

thank you!

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

DataCap allocation requested

512TiB

Id

c4846836-9d89-4281-8b8c-e5afc3ec915c

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

Rule to calculate the allocation request amount

100% weekly > 0.5PiB, requesting 0.5PiB

DataCap allocation requested

512TiB

Total DataCap granted for client so far

512TiB

Datacap to be granted to reach the total amount requested by the client (15PiB)

14.5PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
0 0 512TiB NaN 512TiB
FroghubMan commented 1 year ago

We're preparing dataset.

github-actions[bot] commented 12 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

FroghubMan commented 12 months ago

We're preparing dataset.

filplus-checker-app[bot] commented 12 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

⚠️ 1 storage providers sealed more than 70% of total datacap - f02211244: 100.00%

⚠️ All storage providers are located in the same region.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

xiaoyuaiheshui commented 12 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 12 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

1ane-1 commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

Chuangshi1 commented 11 months ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecnykjk3apekqbeycddlp6ogxbi4maefu4aep6skenhfs6fzzhshy

Address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

Datacap Allocated

512.00TiB

Signer Address

f1mdk7s2vntzm6hu35yuo6vjubtrpfnb2awhgvrri

Id

c4846836-9d89-4281-8b8c-e5afc3ec915c

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecnykjk3apekqbeycddlp6ogxbi4maefu4aep6skenhfs6fzzhshy

AlanGreaterheat commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

AlanGreaterheat commented 11 months ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaced4vdvxwbjrbj4puepcehyn5mbjvp75qgwm6efgsnpwmbduvezhzg

Address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

Datacap Allocated

512.00TiB

Signer Address

f1pnmzlxj7cfeo2v6oj5nco46hkg2l46wj7o4xxui

Id

c4846836-9d89-4281-8b8c-e5afc3ec915c

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaced4vdvxwbjrbj4puepcehyn5mbjvp75qgwm6efgsnpwmbduvezhzg

FroghubMan commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

large-datacap-requests[bot] commented 11 months ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

DataCap allocation requested

1PiB

Id

5457863c-7964-408e-8b66-bcc14d28d6e9

FroghubMan commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

luobin544 commented 11 months ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebyvlz6iwfd65nmm2owyylho5yftblndsgxhblvhrvf6nqxtticiu

Address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

Datacap Allocated

1.00PiB

Signer Address

f1tbd632f6w62glfaf7wjpimacbnjiz26poyoes2q

Id

5457863c-7964-408e-8b66-bcc14d28d6e9

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebyvlz6iwfd65nmm2owyylho5yftblndsgxhblvhrvf6nqxtticiu

newwebgroup commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

newwebgroup commented 11 months ago

The inspection report looks fine, there is no CID sharing.

Regarding the 200T data, the distribution of data copies does not comply with the rules. Can you provide a list of improvement plans for follow-up?

image
FroghubMan commented 11 months ago
image

We are proceeding according to the plan, but the disk numbers allocated to f02364899, f02636860, and f02810687 are at the back, and their packaging speed is relatively fast. In the next round, other nodes will catch up!

newwebgroup commented 11 months ago

When I was signing this LDN, I encountered some errors.

After logging in again to Filplus and plugging and unplugging the Ledger, the error still persists. It needs to be handled by the governance team. @Kevin-FF-USA @simonkim0515

image
nj-steve commented 11 months ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceachsdhkg5gdufj2vbf7gzhofbtpheqk7kwlmg3aioexmnckk63t6

Address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

Datacap Allocated

1.00PiB

Signer Address

f1xx6555qijma7igpnjspyvdunc4vfxkawnpqy5ii

Id

5457863c-7964-408e-8b66-bcc14d28d6e9

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceachsdhkg5gdufj2vbf7gzhofbtpheqk7kwlmg3aioexmnckk63t6

nj-steve commented 11 months ago

The client has explained the 200T data copies.

large-datacap-requests[bot] commented 11 months ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f02049625

Client address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

DataCap allocation requested

2PiB

Id

5cf03a86-ce3f-4069-b405-e4ee68e6a2e5

FroghubMan commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 37.13% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard.

AlanGreaterheat commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 35.29% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard.

AlanGreaterheat commented 11 months ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceazuffgrisotxowm3y44zyer6uxfyevw3zzcgohi2yjb7n46lht4q

Address

f1h2neemc5yldsckegrvyo5dq26ybo2i66ukgqsei

Datacap Allocated

2.00PiB

Signer Address

f1pnmzlxj7cfeo2v6oj5nco46hkg2l46wj7o4xxui

Id

5cf03a86-ce3f-4069-b405-e4ee68e6a2e5

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceazuffgrisotxowm3y44zyer6uxfyevw3zzcgohi2yjb7n46lht4q

NiwanDao commented 11 months ago

LGTM