search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
109 stars 62 forks source link

[DataCap Application] Public Datasets --- NOAA #2112

Closed FroghubMan closed 11 months ago

FroghubMan commented 1 year ago

Data Owner Name

NOAA; 4DN-DCIC; NWGSC; NRC; UBC

What is your role related to the dataset

Data Preparer

Data Owner Country/Region

United States

Data Owner Industry

Other

Website

http://www.noaa.gov/

Social Media

https://twitter.com/NOAA
https://twitter.com/4dn_dcic
https://www.facebook.com/EnvironmentandNaturalResourcesinCanada/
https://twitter.com/NRCan
https://www.youtube.com/user/NaturalResourcesCa

Total amount of DataCap being requested

8PiB

Expected size of single dataset (one copy)

1

Number of replicas to store

8

Weekly allocation of DataCap requested

1PiB

On-chain address for first allocation

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Identifier

No response

Share a brief history of your project and organization

FrogHub has always defined itself as a tool developer and infrastructure builder in the Filecoin ecosystem. In 2019, we started to focus on technical solutions and development based on the IPFS protocol and Filecoin network. We have been working hard to become a qualified builder in the filecoin ecosystem.

Our team is a very pure development team, more than 90% of which are developers, more than half of whom have more than 5 years of development experience in communication, Internet, blockchain and other industries. We hope that we can gain users' recognition by exporting useful tools and platforms.

In order to contribute to the filecoin community, we have developed the open source sector repair tool Filecoin-Sealer-Recover and the nft free authoring platform NFT-Creator.
In addition, we plan to provide a sector browser for the community in 2023 and build the liquidity pledge platform STFIL on FVM.

See the links below for details.
- making: https://github.com/orgs/froghub-io

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

NOAA Coastal Lidar Dataset
Released and archived 4DNucleome data
Nanopore Reference Human Genome
NRT Sentinel data in an S3 bucket broken down by sensor, product type and date.
UBC Sunflower Genome Data 1

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

No response

If you are a data preparer. What is your location (City and Country)

No response

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

No response

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

No response

Please share a sample of the data

https://registry.opendata.aws/noaa-coastal-lidar/
https://registry.opendata.aws/4dnucleome/
https://registry.opendata.aws/nanopore/
https://registry.opendata.aws/sentinel-products-ca-mirror/
https://registry.opendata.aws/ubc-sunflower-genome/

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

2 to 3 years

In which geographies do you plan on making storage deals

Greater China, North America, Europe

How will you be distributing your data to storage providers

Cloud storage (i.e. S3), Shipping hard drives

How do you plan to choose storage providers

Slack, Big Data Exchange, Partners

If you answered "Others" in the previous question, what is the tool or platform you plan to use

No response

If you already have a list of storage providers to work with, fill out their names and provider IDs below

No response

How do you plan to make deals to your storage providers

Boost client

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

Sunnyiscoming commented 1 year ago

Have you prepared enough token for sector pledge? Why are you choosing to store it again? Best practice for storing large datasets includes ideally, storing it in 3 or more regions, with 4 or more storage provider operators or owners.You should list Miner ID, Business Entity, Location of sps you will cooperate with. Could you send an email to [filplus-app-review@fil.org] with your official domain in order to confirm your identity? Email name should includes the issue id #2112 .

FroghubMan commented 1 year ago

1.I am a data preparer, and the SP I am working with has enough FIL as storage collateral. 2.We want to store valuable data. The data for this application is something we have not stored before. 3.The following are the SPs we have cooperated with.

MinerID City Continent
f01811024 HongKong CN
f0827006 Tokyo Japan
f02227726 HongKong CN
f02195153 Tokyo Japan
f02098006 Krabi Thailand
f02223012 Hebei CN
f02235154 HongKong CN
f02085722 Jiangmen CN

4.I have sent the confirmation email.

FroghubMan commented 1 year ago

Hi @Filplus-govteam @Sunnyiscoming ,I have submitted the information in #2119, please review it for me. Thank you very much.

Sunnyiscoming commented 1 year ago

Datacap Request Trigger

Total DataCap requested

8PiB

Expected weekly DataCap usage rate

1PiB

Client address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

DataCap allocation requested

409.59TiB

Id

365d202a-3ba0-4fe2-9e46-81a7ea38149b

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

FroghubMan commented 1 year ago

We are preparing the data.

FroghubMan commented 1 year ago

Hi @Sunnyiscoming ,We have been preparing data, only to find that it was closed. We have now completed the prep function, can you reopen this application for me?

1ane-1 commented 1 year ago

Could you send an email to [filplus-app-review@fil.org] with your official domain in order to confirm your identity? Email name should includes the issue id https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/2112 . Could you show your email screenshot above that for us to confirm your identity? Thanks. @FroghubMan

FroghubMan commented 1 year ago

Screenshot_20230831_100248_Gmail

Chuangshi1 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceck7huapkgbfo243yjhpp5lqmlcjvtaji5hp73iutmunomh22a45m

Address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

Datacap Allocated

409.59TiB

Signer Address

f1mdk7s2vntzm6hu35yuo6vjubtrpfnb2awhgvrri

Id

365d202a-3ba0-4fe2-9e46-81a7ea38149b

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceck7huapkgbfo243yjhpp5lqmlcjvtaji5hp73iutmunomh22a45m

ipollo00 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceclulf24bcol3vqp2fp5nbhqeoeowpyeh2yqrpskvygmwsnjtsyhi

Address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

Datacap Allocated

409.59TiB

Signer Address

f1n5wlrrhoxpkgwij25xrtt7w7g2k3fhbthmdn6ri

Id

365d202a-3ba0-4fe2-9e46-81a7ea38149b

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceclulf24bcol3vqp2fp5nbhqeoeowpyeh2yqrpskvygmwsnjtsyhi

FroghubMan commented 1 year ago

thank you!

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

DataCap allocation requested

512TiB

Id

b00920f9-bc9e-4d65-95e0-d9cabca77333

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

Rule to calculate the allocation request amount

100% weekly > 0.5PiB, requesting 0.5PiB

DataCap allocation requested

512TiB

Total DataCap granted for client so far

409.59TiB

Datacap to be granted to reach the total amount requested by the client (8PiB)

7.60PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
0 0 409.59TiB NaN 409.59TiB
FroghubMan commented 1 year ago

We're preparing dataset.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

FroghubMan commented 1 year ago

We're preparing dataset.

1ane-1 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 73.72% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

sxxfuture-official commented 1 year ago

Others look good, please increase the number of data replicas in the subsequent process.

sxxfuture-official commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecmsifh3imabr4tosqghhx72ohewonhxobhjp3innel4w2wq6u2sw

Address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

Datacap Allocated

512.00TiB

Signer Address

f1foiomqlmoshpuxm6aie4xysffqezkjnokgwcecq

Id

b00920f9-bc9e-4d65-95e0-d9cabca77333

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecmsifh3imabr4tosqghhx72ohewonhxobhjp3innel4w2wq6u2sw

Bitrise0111 commented 1 year ago

Check report shows healthy and we'd like to support. We recommand there should be more data backup.

Bitrise0111 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceb3joewbxwg5ox5sf6i2ieoezg2f7mc4h5rao4h2fi4tox77hwyea

Address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

Datacap Allocated

512.00TiB

Signer Address

f1nknj7ayq4o43czrtdoauggtwl43fbqatmqis3yy

Id

b00920f9-bc9e-4d65-95e0-d9cabca77333

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceb3joewbxwg5ox5sf6i2ieoezg2f7mc4h5rao4h2fi4tox77hwyea

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

FroghubMan commented 1 year ago

We're preparing data

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

DataCap allocation requested

1PiB

Id

fba96cbc-8cc2-44fe-a096-00c9600f4c7a

FroghubMan commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

1ane-1 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

1ane-1 commented 1 year ago

Report is ok, willing to support this round.

1ane-1 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedafhxisvvzpw7dbywz7oa3piux7344cvv6whxe2usho3bbzqygr6

Address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

Datacap Allocated

1.00PiB

Signer Address

f1mdk7s2vntzm6hu35yuo6vjubtrpfnb2awhgvrri

Id

fba96cbc-8cc2-44fe-a096-00c9600f4c7a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedafhxisvvzpw7dbywz7oa3piux7344cvv6whxe2usho3bbzqygr6

laurarenpanda commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedqtic7dsjwl4trrmhy2iyyr4tyzvwcwzpajhxs3vyqelvghwvxiq

Address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

Datacap Allocated

1.00PiB

Signer Address

f1bp3tzp536edm7dodldceekzbsx7zcy7hdfg6uzq

Id

fba96cbc-8cc2-44fe-a096-00c9600f4c7a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedqtic7dsjwl4trrmhy2iyyr4tyzvwcwzpajhxs3vyqelvghwvxiq

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

DataCap allocation requested

1PiB

Id

194ec683-98c0-4466-aa69-31350eca98ea

FroghubMan commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

SuperChaiChai commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebycrjeqtrvvcvlycricdtfx5jbil7iplfkjl2vz5ykiqcxyewbp4

Address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

Datacap Allocated

1.00PiB

Signer Address

f12mckci3omexgzoeosjvstcfxfe4vqw7owdia3da

Id

194ec683-98c0-4466-aa69-31350eca98ea

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebycrjeqtrvvcvlycricdtfx5jbil7iplfkjl2vz5ykiqcxyewbp4

kevzak commented 1 year ago

SPs taking deals

f0870354 | Beijing, Beijing, CNChina Unicom Beijing Province Network | 96.07 TiB | 10.57% | 96.07 TiB | 0.00% f01989372 | Beijing, Beijing, CNChina Unicom Beijing Province Network | 96.04 TiB | 10.57% | 96.04 TiB | 0.00% f02816837 | Shenzhen, Guangdong, CNChina Unicom Shenzen network | 29.91 TiB | 3.29% | 29.91 TiB | 0.00% f01907578 | Xiamen, Fujian, CNCHINANET-BACKBONE | 48.22 TiB | 5.31% | 48.22 TiB | 0.00% f0123931 | Xiamen, Fujian, CNFuzhou | 80.00 TiB | 8.80% | 80.00 TiB | 0.00% f02636860 | Hong Kong, Central and Western, HKHKBN Enterprise Solutions HK Limited | 175.86 TiB | 19.35% | 175.86 TiB | 0.00% f02810687 | Hong Kong, Central and Western, HKHKBN Enterprise Solutions HK Limited | 170.94 TiB | 18.81% | 170.94 TiB | 0.00% f01854510new | Hong Kong, Central and Western, HKKaopu Cloud HK Limited | 93.03 TiB | 10.24% | 93.03 TiB | 0.00% f02806894 | Paripark, Seoul, KRThe Constant Company, LLC | 50.78 TiB | 5.59% | 50.78 TiB | 0.00% f02372022 | Ōi, Saitama, JPThe Constant Company, LLC | 49.45 TiB | 5.44% | 49.45 TiB | 0.00% f02803754 | Osaka, Ōsaka, JPThe Constant Company, LLC | 18.39 TiB | 2.02% | 18.39 TiB | 0.00%

Original list provided:

f01811024 | HongKong | CN f0827006 | Tokyo | Japan f02227726 | HongKong | CN f02195153 | Tokyo | Japan f02098006 | Krabi | Thailand f02223012 | Hebei | CN f02235154 | HongKong | CN f02085722 | Jiangmen | CN

@FroghubMan Not seeing any of the original SP IDs matching here, please explain.

FroghubMan commented 1 year ago

@kevzak Thank you for your attention. The actual allocated miner node is different from the miner node during KYC because if the data preparation time is long, some SPs will first store other available datacaps for the sake of equipment usage efficiency. When the storage power of the miner node reaches a certain value, SPs will consider creating a new miner node. Therefore, over time, changes in the miner node that stores datacap are inevitable.

However, we ensure that everything strictly abides by the rules of FIL+, and we can complete KYC for new nodes if necessary.

kevzak commented 1 year ago

Please provide as much information as you can about actual Miner IDs being used and their entities being different and distributed.

FroghubMan commented 1 year ago

Please provide as much information as you can about actual Miner IDs being used and their entities being different and distributed.

Sure.

MinerID Entity City Continent
f0870354 Personal beijing CN
f01989372 Personal beijing CN
f02816837 B.B Tech guangdong CN
f01907578 ssdminer Putian CN
f0123931 ssdminer Fuzhou CN
f02806894 Chimsen Seoul Korea
f02372022 Chimsen Tokyo Japan
f02810687 DR66 Tech HongKong CN
f02636860 DR66 Tech HongKong CN
f01854510 3cloud tech HongKong CN
f02803754 Personal Osaka Japan
NiwanDao commented 1 year ago

SP Distribution plan is reasonable. I will support for this round.

NiwanDao commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedrfy6q7nokoxevvy3ddgjwc6edeca6oqb4dqliqjf6jplwmifmzo

Address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

Datacap Allocated

1.00PiB

Signer Address

f1a2lia2cwwekeubwo4nppt4v4vebxs2frozarz3q

Id

194ec683-98c0-4466-aa69-31350eca98ea

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedrfy6q7nokoxevvy3ddgjwc6edeca6oqb4dqliqjf6jplwmifmzo

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f02049625

Client address

f1svst5e6a7b2ux2scyc336mkmdgec3eqthbjtjpa

DataCap allocation requested

2PiB

Id

f5c9f959-d4ce-4686-8134-af13e48fb786

FroghubMan commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard.

1ane-1 commented 1 year ago

checker:manualTrigger