search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
109 stars 62 forks source link

[DataCap Application] <FogMeta Lab> - <Sentinel-2 Cloud-Optimized GeoTIFFs V1> #2286

Open hengdingy opened 11 months ago

hengdingy commented 11 months ago

Data Owner Name

FogMeta Lab

What is your role related to the dataset

Data Preparer

Data Owner Country/Region

China

Data Owner Industry

Web3 / Crypto

Website

https://fogmetalabs.com

Social Media

Twitter: https://twitter.com/FogMeta
GitHub: https://github.com/FogMeta

Total amount of DataCap being requested

15PiB

Expected size of single dataset (one copy)

16.5 PiB

Number of replicas to store

4

Weekly allocation of DataCap requested

1PiB

On-chain address for first allocation

f1lgh5chkslwjql2fekxs3dflc6aznfkpexf5awfa

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Identifier

No response

Share a brief history of your project and organization

FogMeta Lab's research spans multiple levels from system technology, infrastructure, and middleware to services and solutions, and involves future systems, network technology and business, distributed systems and management, information management, and interactive and innovative services. Based on the views on and practices in the industry, FogMeta also solves the problem of business complexity through operations optimization and other technologies.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

"The Sentinel-2 mission is a land monitoring constellation of two satellites that provide high resolution optical imagery and provide continuity for the current SPOT and Landsat missions. The mission provides a global coverage of the Earth's land surface every 5 days, making the data of great use in ongoing studies. This dataset is the same as the Sentinel-2 dataset, except the JP2K files were converted into Cloud-Optimized GeoTIFFs (COGs). Additionally, SpatioTemporal Asset Catalog metadata has were in a JSON file alongside the data, and a STAC API called Earth-search is freely available to search the archive. This dataset contains all of the scenes in the original Sentinel-2 Public Dataset and will grow as that does. L2A data are available from April 2017 over wider Europe region and globally since December 2018.

Update Frequency
New Sentinel data are added regularly, usually within few hours after they are available on Copernicus OpenHub."

Source: https://registry.opendata.aws/sentinel-2-l2a-cogs/

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

No response

If you are a data preparer. What is your location (Country/Region)

China

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

IPFS, lotus, graphsplit, others/custom tool, MetaArk(https://fogmetalabs.com/metaark)

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

We are THE FIRST data preparers that works on this dataset.

Please share a sample of the data

s3://sentinel-cogs/(16.4 PiB)
s3://sentinel-cogs-inventory/(3.4 TiB)

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Monthly

For how long do you plan to keep this dataset stored on Filecoin

2 to 3 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, Africa, North America, South America, Europe, Australia (continent), Antarctica

How will you be distributing your data to storage providers

Cloud storage (i.e. S3), HTTP or FTP server, IPFS, Shipping hard drives, Others

How do you plan to choose storage providers

Slack, Partners, Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

We'd also like to use Swan Chain (https://www.swanchain.io/) to choose storage providers who meet our requirements.

If you already have a list of storage providers to work with, fill out their names and provider IDs below

The storage providers we'd like to work with are presented below. Some of them are from the FilSwan platform.
f01955033
f02029115
f03624
f010088
f02301
f08399
f02401
f01955030
f0187709
f01163272
f01402814
f01390330
f01225882
f0717969
f03223
f01395673
f01072221
f0143858
f01786736
f0836160
f032824
f01443744
f01871352
f01907556
f01955028
f01947280
f01946551
f02012951
f01970630
f0240185

How do you plan to make deals to your storage providers

Boost client, Lotus client, Others/custom tool

If you answered "Others/custom tool" in the previous question, enter the details here

Swan Client tool
https://github.com/filswan/go-swan-client

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 11 months ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

hengdingy commented 11 months ago

The new large dataset application will serve as our primary application moving forward. Consequently, all previously submitted applications have been closed, including: https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1739 https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1738; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1740; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1736; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1735; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1734; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1733; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1737; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1741; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1742

herrehesse commented 11 months ago

Great! Much more clarity.

ghost commented 11 months ago

@hengdingy 16.5PiB x 4 = 66PiB correct? you are applying for 15

ghost commented 11 months ago

Also can you validate the SPs being used, entity and locations. How does Filswan collect and verify this information?

hengdingy commented 11 months ago

@hengdingy 16.5PiB x 4 = 66PiB correct? you are applying for 15

yes, we think 15PiB is a big amount datacap for us to prepare and send the data to the SPs in the first phase. After the phase, we will continue to apply the second datacap.

hengdingy commented 11 months ago

Also can you validate the SPs being used, entity and locations. How does Filswan collect and verify this information?

@Filplus-govteam Swan platform is a public platform, SPs can join it by validate the owner address signature. And Swan platform will auto-match and distribute the data to SPs by query-ask, fast-retrieve requirements and locations; for now, A part of the SPs information can be found here. And considering the platform's user privacy, we just give the permission to the fil+ gov team to verify. Please apply for the form's permission

hengdingy commented 11 months ago

@kevzak you have got the permission of the form

kevzak commented 11 months ago

@hengdingy thank you, it was reviewed. @Sunnyiscoming please trigger and we'll review deal onboarding

Sunnyiscoming commented 11 months ago

Datacap Request Trigger

Total DataCap requested

15PiB

Expected weekly DataCap usage rate

1PiB

Client address

f1lgh5chkslwjql2fekxs3dflc6aznfkpexf5awfa

large-datacap-requests[bot] commented 11 months ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1lgh5chkslwjql2fekxs3dflc6aznfkpexf5awfa

DataCap allocation requested

512TiB

Id

a103274b-d991-4323-bd58-4089a1ae6317

Sunnyiscoming commented 11 months ago

Please add provide ID, City, Country, Organization of each SP here according with the format https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/2281#issuecomment-1833941809

cryptoAmandaL commented 11 months ago

@hengdingy, thanks to Filswan for the contributions to the community. I've noticed you've been active in the Fil+ community for a while, and you have rich experience in dealing with large amounts of data. For instance, this time you mentioned expecting more than 60 PiB.

I would like to ask, as a senior member of the community, if you could share any insights regarding how you verify the consistency of customer-stored data. What methods do you use for tracking them? This question has been discussed in the community for a long time, and I believe the @Filplus-govteam would also be interested in hearing from experienced users like yourself.

Thank you. :)

AlanGreaterheat commented 11 months ago

Nice community project, willing to support.

For the SPs location distribution, the applicant told me the verification method via slack, looking forward to see the follow-up progress.

AlanGreaterheat commented 11 months ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceaam7i4aupsgnoin6qhgzusw35fopbvls6bd3o5uazktxbkdsvlck

Address

f1lgh5chkslwjql2fekxs3dflc6aznfkpexf5awfa

Datacap Allocated

512.00TiB

Signer Address

f1pnmzlxj7cfeo2v6oj5nco46hkg2l46wj7o4xxui

Id

a103274b-d991-4323-bd58-4089a1ae6317

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceaam7i4aupsgnoin6qhgzusw35fopbvls6bd3o5uazktxbkdsvlck

SuperChaiChai commented 11 months ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedwtufjom332wjhrdlxqxvk3oevnluukm3fbieznjhmf7wzaytuss

Address

f1lgh5chkslwjql2fekxs3dflc6aznfkpexf5awfa

Datacap Allocated

512.00TiB

Signer Address

f12mckci3omexgzoeosjvstcfxfe4vqw7owdia3da

Id

a103274b-d991-4323-bd58-4089a1ae6317

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedwtufjom332wjhrdlxqxvk3oevnluukm3fbieznjhmf7wzaytuss

hengdingy commented 11 months ago

Please add provide ID, City, Country, Organization of each SP here according with the format #2281 (comment)

https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1999#issuecomment-1824684456

@Sunnyiscoming please request access to the form)

hengdingy commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

Sunnyiscoming commented 11 months ago

I have ask for access to the form. I will list the information of these nodes here.

github-actions[bot] commented 11 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

hengdingy commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

hengdingy commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

Sunnyiscoming commented 10 months ago

There is no organization name of each nodes. Can you add them in the form? @hengdingy

hengdingy commented 10 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 10 months ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

hengdingy commented 10 months ago

checker:manualTrigger

DataCap and CID Checker Report1

No active deals found for this client.

Footnotes

  1. To manually trigger this report, add a comment with text checker:manualTrigger ↩

checker:manualTrigger

filplus-checker-app[bot] commented 10 months ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

github-actions[bot] commented 10 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

hengdingy commented 10 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 10 months ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

github-actions[bot] commented 9 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

hengdingy commented 9 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 9 months ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

hengdingy commented 9 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 9 months ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

github-actions[bot] commented 9 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

hengdingy commented 9 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 9 months ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

hengdingy commented 9 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 9 months ago

DataCap and CID Checker Report[^1]

No active deals found for this client.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

hengdingy commented 9 months ago

DataCap and CID Checker Report1

No active deals found for this client.

Footnotes

  1. To manually trigger this report, add a comment with text checker:manualTrigger ↩

checker:manualTrigger

github-actions[bot] commented 9 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

Kirameta commented 9 months ago

manualTrigger

github-actions[bot] commented 8 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

hengdingy commented 8 months ago

manualTrigger

github-actions[bot] commented 8 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

hengdingy commented 8 months ago

manualTrigger