filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] JMA Himawari-8/9 #1860

Closed Flora-Moody closed 1 year ago

Flora-Moody commented 1 year ago

Data Owner Name

NOAA

Data Owner Country/Region

Japan

Data Owner Industry

Environment

Website

https://www.noaa.gov/

Social Media

https://www.facebook.com/NOAA
https://www.youtube.com/usnoaagov

Total amount of DataCap being requested

5PiB

Weekly allocation of DataCap requested

1PiB

On-chain address for first allocation

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Identifier

No response

Share a brief history of your project and organization

NOAA is an agency that enriches life through science. Our reach goes from the surface of the sun to the depths of the ocean floor as we work to keep the public informed of the changing environment around them.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

Himawari-8, stationed at 140E, owned and operated by the Japan Meteorological Agency (JMA), is a geostationary meteorological satellite, with Himawari-9 as on-orbit back-up, that provides constant and uniform coverage of east Asia, and the west and central Pacific regions from around 35,800 km above the equator with an orbit corresponding to the period of the earth’s rotation.

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

No response

How do you plan to prepare the dataset

IPFS, lotus, singularity

If you answered "other/custom tool" in the previous question, enter the details here

No response

Please share a sample of the data

https://registry.opendata.aws/noaa-himawari/

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

1.5 to 2 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, North America, Europe

How will you be distributing your data to storage providers

Cloud storage (i.e. S3), HTTP or FTP server, Shipping hard drives

How do you plan to choose storage providers

Slack, Filmine, Big data exchange

If you answered "Others" in the previous question, what is the tool or platform you plan to use

No response

If you already have a list of storage providers to work with, fill out their names and provider IDs below

No response

How do you plan to make deals to your storage providers

Boost client, Lotus client

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

Sunnyiscoming commented 1 year ago

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

1PiB

Client address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

DataCap allocation requested

256TiB

Id

5772f50d-4db3-4f43-a9f3-a8f16e319bb6

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

DataCap allocation requested

256TiB

Id

02382fad-8c02-4613-a308-cc71f01feeed

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

No application info found for this issue on https://filplus.d.interplanetary.one/clients.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

No application info found for this issue on https://filplus.d.interplanetary.one/clients.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

Casey-PG commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceagod25pffke7uaja7kjzu7mjdwsfwlja5hpi56dkdwr7hkasjujw

Address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

Datacap Allocated

256.00TiB

Signer Address

f1d4yb3wags3mtddzesxoo63jv7dmlec3bq4yteni

Id

02382fad-8c02-4613-a308-cc71f01feeed

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceagod25pffke7uaja7kjzu7mjdwsfwlja5hpi56dkdwr7hkasjujw

BobbyChoii commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedsxvmfppszgawgtsx63zknyz2ze5kjzal5hdyhocz3ymebsoznpq

Address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

Datacap Allocated

256.00TiB

Signer Address

f1irqs2gmctiv3jcdfwuch7oxvf4ixh3k4b2wc24i

Id

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedsxvmfppszgawgtsx63zknyz2ze5kjzal5hdyhocz3ymebsoznpq

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

DataCap allocation requested

512TiB

Id

db11a179-6403-461a-954c-a8dd6b13b157

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

Rule to calculate the allocation request amount

10% of total dc amount requested

DataCap allocation requested

512TiB

Total DataCap granted for client so far

256TiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

4.75PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
null null 256TiB null 64.28TiB
filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

cryptowhizzard commented 1 year ago

Hello,

I checked on this application but you are not following on the rules of distribution as everything is stored in China. Can you tell us why this is and how you are going to fix this?

Scherm­afbeelding 2023-04-12 om 15 28 20

@Casey-PG Can you let us (the community) know what due diligence has been done here for this client? What is the data onboarding plan for this client? Where is he/she going to store the data and who is she? What is their internet capacity? On what bases have you approved this application and why?

TakiChain commented 1 year ago

Checked the information applicant shared and so far everything is in line with Filplus program rules. Willing to give my signature in the next round. Keep up the good work!

TakiChain commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecuxjwvlcfn43o27lfk42evquga3gwztpphmhnvwybjxgpgvvfiyg

Address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

Datacap Allocated

512.00TiB

Signer Address

f15impf3j2zcaex4lhyxndxswuuhv24vzstuqtxsi

Id

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecuxjwvlcfn43o27lfk42evquga3gwztpphmhnvwybjxgpgvvfiyg

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 58.11% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

Bennyyangpu commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedtn5sg4fihcrunxghudx4sjogrmh2wnla66m74svojgu4rxzbcuy

Address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

Datacap Allocated

512.00TiB

Signer Address

f174fg3bqbln3zjnkxtyf6s54txqkr7yqkj6cig7y

Id

db11a179-6403-461a-954c-a8dd6b13b157

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedtn5sg4fihcrunxghudx4sjogrmh2wnla66m74svojgu4rxzbcuy

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

DataCap allocation requested

1PiB

Id

8ed8b509-d2da-4571-bbd8-59c987a2abf8

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

Rule to calculate the allocation request amount

20% of total dc amount requested

DataCap allocation requested

1PiB

Total DataCap granted for client so far

465661.3YiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

-5.62B

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
16353 11 512TiB 24.05 178.71TiB
MEIYAN666 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacea2enc6ulyuyhohiwdol7k7grjqofyhub7t3g7suaao5dt6b2eifm

Address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

Datacap Allocated

1.00PiB

Signer Address

f1bwugfihrmn3iyunzyxst5nttql3dge4khwmurtq

Id

8ed8b509-d2da-4571-bbd8-59c987a2abf8

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea2enc6ulyuyhohiwdol7k7grjqofyhub7t3g7suaao5dt6b2eifm

AthSmith commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebt26slvxiplz7paqfhzzomd6bwpmvvfd74qs5234bchojfvdwzgg

Address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

Datacap Allocated

1.00PiB

Signer Address

f1vxbqrf7rfum3n6m5u6eb4re6xj7amvsaqnzu64y

Id

8ed8b509-d2da-4571-bbd8-59c987a2abf8

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebt26slvxiplz7paqfhzzomd6bwpmvvfd74qs5234bchojfvdwzgg

AthSmith commented 1 year ago

willing to support this round and keep an eye on the upcoming report.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f02049625

Client address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

DataCap allocation requested

2PiB

Id

b2fc6540-0613-474e-b83e-55a03bf8e353

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

Rule to calculate the allocation request amount

40% of total dc amount requested

DataCap allocation requested

2PiB

Total DataCap granted for client so far

931322574615478927360.0YiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

-1.12B

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
33555 16 1PiB 20.38 261.25TiB
cryptowhizzard commented 1 year ago

Hi @Flora-Moody

It seems none of your SP's are supporting retrieval. This is against FIL+ guidelines. Secondly it seems you don't distribute outside your region ( Asia ) . This is also against FIL+ guidelines.

Please contact your SP's to allow retrieval as soon as possible.

Scherm­afbeelding 2023-05-05 om 20 08 42
Casey-PG commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 35.68% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

Casey-PG commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceb4bzongngggi4axwrp4ey7oimkekmhcxhucqrdkqea5xhfxlq7wi

Address

f1u3gc4fsh47pvrszv3otngb2t5hm3ycmtx7jxxny

Datacap Allocated

2.00PiB

Signer Address

f1d4yb3wags3mtddzesxoo63jv7dmlec3bq4yteni

Id

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceb4bzongngggi4axwrp4ey7oimkekmhcxhucqrdkqea5xhfxlq7wi

Casey-PG commented 1 year ago

First of all, thanks @cryptowhizzard for pointing out the unretrievable data problem. Before addressing your concerns, I would like to state that as a media company we do our best to judge from the description of the application, the information shared by the client combined with the Filplus large dataset project concept. However, unfortunately, our technical field has certain limitations.

We signed the first and fourth rounds of this issue. The signature is provided for the following reasons: (1) This data is publicly available and we recognize its value to meet the project requirements. (2) This client contacted me on slack about their distribution plan. It was also informed that due to the different speeds of sp efficiency, the multi-regional allocation suggested by the project may not be achieved in the early stage. We also find this in many other applications, so we consider it acceptable.

At the Hongkong Web 3 Conference, we deeply felt the importance of clients, data preparers and service providers in ecology. What data means to Filecoin is clear to all participants. The way the notary public judges the project is different, which is a benefit of community diversity and perspectives. We'll pay more attention to retrieval and enhance technical understanding.

Thank you.

cryptowhizzard commented 1 year ago

Hi @Casey-PG

Thanks for your explanation. I understand that not everyone is on the same speed and same track and this is nothing personal. But when a notary indicates that the data is not retrievable / an application is not according to Fil+ rules the others should be wary to sign unless there is clear understanding that things will get on track in the near future.

If we do work here to check everything and do due diligence we expect the others to consider it and wait until discovered issues are resolved. Otherwise we can just go ahead and sign everything without any due diligence.

Flora-Moody commented 1 year ago

As far as I understand, filplus does not have a requirement for data to be scattered across multiple continents. China is a very large country, please refer to my sp distribution table, different cities under different provinces and different regions can meet decentralized storage.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

Flora-Moody commented 1 year ago

ok. Got it.

Joss-Hua commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 30.75% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

Flora-Moody commented 1 year ago

yes

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

Flora-Moody commented 1 year ago

Yes. Please open it

cryptowhizzard commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 30.75% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

cryptowhizzard commented 1 year ago

Yes. Please open it

To what ends?

None of your data is retrievable, that is against FIL+ guidelines.

This application should be closed and datacap should be revoked.

spaceT9 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 33.46% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

ghost commented 1 year ago

Hello @Flora-Moody per the https://github.com/filecoin-project/notary-governance/issues/922 for Open, Public Dataset applicants, please complete the following Fil+ registration form to identify yourself as the applicant and also please add the contact information of the SP entities you are working with to store copies of the data.

This information will be reviewed by Fil+ Governance team to confirm validity and then the application will be approved for notary review. Let us know if you have any questions.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

Flora-Moody commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 35.36% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

spaceT9 commented 1 year ago

checker:manualTrigger