[DataCap Application] <FogMeta Lab> - <Sentinel-2 Cloud-Optimized GeoTIFFs V1>[1/10]

hengdingy commented 1 year ago

Data Owner Name

FogMeta Lab

Data Owner Country/Region

China

Data Owner Industry

Web3 / Crypto

Website

https://fogmeta.com

Social Media

Twitter: https://twitter.com/FogMeta
GitHub: https://github.com/FogMeta

Total amount of DataCap being requested

5PiB

Weekly allocation of DataCap requested

1PiB

On-chain address for first allocation

f1cdg6s3ebuifojokvg3hs5xj4dgvkqnaes5lvmoy

Custom multisig

[ ] Use Custom Multisig

Identifier

No response

Share a brief history of your project and organization

FogMeta Lab's research spans multiple levels from system technology, infrastructure, and middleware to services and solutions, and involves future systems, network technology and business, distributed systems and management, information management, and interactive and innovative services. Based on the views on and practices in the industry, FogMeta also solves the problem of business complexity through operations optimization and other technologies.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

"The Sentinel-2 mission is a land monitoring constellation of two satellites that provide high resolution optical imagery and provide continuity for the current SPOT and Landsat missions. The mission provides a global coverage of the Earth's land surface every 5 days, making the data of great use in ongoing studies. This dataset is the same as the Sentinel-2 dataset, except the JP2K files were converted into Cloud-Optimized GeoTIFFs (COGs). Additionally, SpatioTemporal Asset Catalog metadata has were in a JSON file alongside the data, and a STAC API called Earth-search is freely available to search the archive. This dataset contains all of the scenes in the original Sentinel-2 Public Dataset and will grow as that does. L2A data are available from April 2017 over wider Europe region and globally since December 2018.

Update Frequency
New Sentinel data are added regularly, usually within few hours after they are available on Copernicus OpenHub."

Source: https://registry.opendata.aws/sentinel-2-l2a-cogs/

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

No response

How do you plan to prepare the dataset

IPFS, lotus, graphsplit, others/custom tool

If you answered "other/custom tool" in the previous question, enter the details here

We'd also like to use the Swan Client tool (https://github.com/filswan/go-swan-client#Graphsplit) to prepare the dataset.

Please share a sample of the data

s3://sentinel-cogs/(16.4 PiB)
s3://sentinel-cogs-inventory/(3.4 TiB)

Confirm that this is a public dataset that can be retrieved by anyone on the Network

[X] I confirm

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Monthly

For how long do you plan to keep this dataset stored on Filecoin

2 to 3 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, Africa, North America, South America, Europe, Australia (continent), Antarctica

How will you be distributing your data to storage providers

Cloud storage (i.e. S3), IPFS, Shipping hard drives

How do you plan to choose storage providers

Slack, Partners, Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

We'd also like to use FilSwan platform (https://filswan.com/) to choose storage providers who meet our requirements.

If you already have a list of storage providers to work with, fill out their names and provider IDs below

The storage providers we'd like to work with are presented below. Some of them are from the FilSwan platform.
f01955033
f02029115
f03624
f010088
f02301
f08399
f02401
f01955030
f0187709
f01163272
f01402814
f01390330
f01225882
f0717969
f03223
f01395673
f01072221
f0143858
f01786736
f0836160
f032824
f01443744
f01871352
f01907556
f01955028
f01947280
f01946551
f02012951
f01970630
f0240185

How do you plan to make deals to your storage providers

Boost client, Lotus client, Others/custom tool

If you answered "Others/custom tool" in the previous question, enter the details here

Swan Client tool
https://github.com/filswan/go-swan-client

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

Sunnyiscoming commented 1 year ago

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

1PiB

Client address

f1cdg6s3ebuifojokvg3hs5xj4dgvkqnaes5lvmoy

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1cdg6s3ebuifojokvg3hs5xj4dgvkqnaes5lvmoy

DataCap allocation requested

256TiB

Id

f1fd5bfd-9147-45c5-a91f-f5d76d6d6b2f

cryptowhizzard commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedgrm64gbj6eok6wnd34xcyfwwgcalxie4ku5of6gvlmrcgxcp7di

Address

f1cdg6s3ebuifojokvg3hs5xj4dgvkqnaes5lvmoy

Datacap Allocated

256.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

f1fd5bfd-9147-45c5-a91f-f5d76d6d6b2f

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedgrm64gbj6eok6wnd34xcyfwwgcalxie4ku5of6gvlmrcgxcp7di

nj-steve commented 1 year ago

@hengdingy If all the SPs can be trusted for supporting retrieval by anyone?

hengdingy commented 1 year ago

@nj-steve thanks for your review Yes, all SPs are from FilSwan Platform, and according to their behavior(including fast-retrieval features), SP will get a score, data-deals will be sent to them by Market-matcher. You can see here

hengdingy commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

nj-steve commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedjttz4c5tbncxfiykjtapfapennyk34ww5dynryx5luw3sx5ezt4

Address

f1cdg6s3ebuifojokvg3hs5xj4dgvkqnaes5lvmoy

Datacap Allocated

256.00TiB

Signer Address

f1xx6555qijma7igpnjspyvdunc4vfxkawnpqy5ii

Id

f1fd5bfd-9147-45c5-a91f-f5d76d6d6b2f

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedjttz4c5tbncxfiykjtapfapennyk34ww5dynryx5luw3sx5ezt4

zcfil commented 1 year ago

I have reviewed historical information and am willing to provide support in the first round, and will follow up on the data situation in the later stage.

zcfil commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedue52atbublowjgpkd5xcqkllxfizywguzlqumvg2l5wb25fllbs

Address

f1cdg6s3ebuifojokvg3hs5xj4dgvkqnaes5lvmoy

Datacap Allocated

256.00TiB

Signer Address

f1cjzbiy5xd4ehera4wmbz63pd5ku4oo7g52cldga

Id

f1fd5bfd-9147-45c5-a91f-f5d76d6d6b2f

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedue52atbublowjgpkd5xcqkllxfizywguzlqumvg2l5wb25fllbs

kevzak commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Overall Graphsync retrieval success rate: 14.29%
Overall HTTP retrieval success rate: 0.00%
Overall Bitswap retrieval success rate: 0.00%

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

kevzak commented 1 year ago

Hello @hengdingy is this project still active? What is your role? Data preparer? Or SP? How was the data prepared? How did you find SPs to work with? Where are they located?

kevzak commented 1 year ago

Also we have a new, optional KYC ID check available on filplus.storage. If you want to add additional layer of trust you can complete an ID check and verify this GitHub account. See details: LINK

hengdingy commented 1 year ago

Hi @kevzak. We're the data preparer of this dataset. We use the Swan Client tool to prepare the dataset and send it to SPs. The storage providers are automatically selected by the bidding engine of the FilSwan Platform.

We get the list of providers from the platform. Here is an example. The storage providers' locations and IDs are presented on the page.

kevzak commented 1 year ago

Thank you for the update. This applications CID report https://github.com/data-preservation-programs/filplus-checker-assets/blob/main/filecoin-project/filecoin-plus-large-datasets/issues/1733/1688391279895.md?

Are you saying SP list will grow with future allocations?

hengdingy commented 1 year ago

Yes, @kevzak , the list is a part of all storage providers who will be assigned the deals. So the SP list will grow with future allocations.

hengdingy commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Overall Graphsync retrieval success rate: 3.70%
Overall HTTP retrieval success rate: 0.00%
Overall Bitswap retrieval success rate: 0.00%

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

hengdingy commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Overall Graphsync retrieval success rate: 11.85%
Overall HTTP retrieval success rate: 1.48%
Overall Bitswap retrieval success rate: 0.00%

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

Normalnoise commented 1 year ago

https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1703

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

hengdingy commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Overall Graphsync retrieval success rate: 8.27%
Overall HTTP retrieval success rate: 39.61%
Overall Bitswap retrieval success rate: 1.18%

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

github-actions[bot] commented 12 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

hengdingy commented 12 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 12 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Overall Graphsync retrieval success rate: 6.49%
Overall HTTP retrieval success rate: 46.15%
Overall Bitswap retrieval success rate: 0.90%

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

large-datacap-requests[bot] commented 11 months ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1cdg6s3ebuifojokvg3hs5xj4dgvkqnaes5lvmoy

Rule to calculate the allocation request amount

200% weekly > 1PiB, requesting 1PiB

DataCap allocation requested

1PiB

Total DataCap granted for client so far

232830.6YiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

232830.6YiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
1061	9	256TiB	46.65	478.84TiB

Normalnoise commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Overall Graphsync retrieval success rate: 6.10%
Overall HTTP retrieval success rate: 39.64%
Overall Bitswap retrieval success rate: 0.91%

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

github-actions[bot] commented 11 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

hengdingy commented 11 months ago

keep alive

Normalnoise commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Overall Graphsync retrieval success rate: 5.85%
Overall HTTP retrieval success rate: 36.47%
Overall Bitswap retrieval success rate: 0.95%

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

github-actions[bot] commented 10 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

hengdingy commented 10 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 10 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Overall Graphsync retrieval success rate: 5.73%
Overall HTTP retrieval success rate: 35.01%
Overall Bitswap retrieval success rate: 1.03%

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

github-actions[bot] commented 10 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

Normalnoise commented 10 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 10 months ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard.

herrehesse commented 9 months ago

@hengdingy providers are highly centralised in Asia, not many in other regions. Will this change as you stated in the LDN?

hengdingy commented 9 months ago

Yes, the LDN will be assigned to more regions

github-actions[bot] commented 9 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

Normalnoise commented 9 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 9 months ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard.

Sunnyiscoming commented 9 months ago

Hello, @hengdingy per the https://github.com/filecoin-project/notary-governance/issues/922 for Open, Public Dataset applicants, please complete the following Fil+ registration form to identify yourself as the applicant and also please add the contact information of the SP entities you are working with to store copies of the data.

This information will be reviewed by Fil+ Governance team to confirm validity and then the application will be allowed to move forward for additional notary review.

hengdingy commented 9 months ago

@Sunnyiscoming https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1999#issuecomment-1824684456

filecoin-project / filecoin-plus-large-datasets