search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] Kernelogic - Sentinel3 (2/3) #1107

Closed kernelogic closed 1 year ago

kernelogic commented 1 year ago

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Share a brief history of your project and organization.

I have participated every Slingshot phase and is probably the best performing as a "small individual client". 

Even though Slingshot v2 has ended, there are still strong demand from SPs to onboard useful data. This application is to onboard open dataset from AWS.

I will provide a nice web UI (https://github.com/tech-greedy/singularity-browser) to index all files onboarded and provide ways to retrieve.

I have successfully completed a few LDNs on other datasets and I have record to show I have been following the rules of decentralization and have zero self dealing.

https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/60
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/59
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/46
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/297
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/298
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/304

What is the primary source of funding for this project?

Self-funded, BigD exchange.

What other projects/ecosystem stakeholders is this project associated with?

enterprise-sp-wg, BigD exchange.

Use-case details

Describe the data being stored onto Filecoin

https://github.com/Sentinel-5P/data-on-s3/blob/master/DocsForAws/Sentinel3Description.md

Sentinel-3 is a Copernicus satellite whose main three sensors are:

Ocean and Land Colour Instrument (OLCI) for medium resolution marine and terrestrial optical measurements.
Sea and Land Surface Temperature Radiometer (SLSTR) for thermal measurements(both marine and terrestrial), land monitoring and fire detection.
SAR Radar Altimeter (SRAL) together with the MicroWave Radiometer (MWR) and Precise Orbit Determination (POD) for ocean topography measurements.

Total size: about 1.5PB, this dataset consists several s3 buckets:
arn:aws:s3:::meeo-s3-cog/   eu-central-1    631.2 TiB
arn:aws:s3:::meeo-s3/NTC/   eu-central-1    409.6 TiB
arn:aws:s3:::meeo-s3/NRT/   eu-central-1    315.6 TiB
arn:aws:s3:::meeo-s3/STC/   eu-central-1    27.5 TiB

Where was the data in this dataset sourced from?

https://registry.opendata.aws/sentinel-3/

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this. 

https://registry.opendata.aws/sentinel-3/

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

Yes, see
https://sentinel.esa.int/documents/247904/690755/Sentinel_Data_Legal_Notice

What is the expected retrieval frequency for this data?

Multiple times per year.

For how long do you plan to keep this dataset stored on Filecoin?

18 months.

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

All regions.

How will you be distributing your data to storage providers? Is there an offline data transfer process?

I will upload my prepared CAR files to a web server and coordinate with providers to download and propose offline deals.

Maximum 3 copies per SP entity and maximum of 12 copies for every pieceCID.

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

Beside the previous SPs I have worked with, I also utilize bigD exchange to further decentralize the storage

To name a few from the community that I deal with regularly: PIKNIK, Holon, CabrinaHuang, HarryM, BigBear, j1v, XinAn Xu, WillTechMusing.

From BigD exchange: Mog Li, Devin Chen, DSS Nathanial Marsh, Rabinovitch, Vin K, arockpool Tony

How will you be distributing deals across storage providers?

Evenly across all providers I propose to, if they can handle. If a miner is a notary itself, this notary will receive no more than 20% of the total granted datacap.

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

I have all I need to start making deals.
large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!
large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

raghavrmadya commented 1 year ago

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

750TiB

Client address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f01858410

Client address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

DataCap allocation requested

256TiB

newwebgroup commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebirygsqvde5blaz356gclscvjih52vgis5qjkj5upqicvagcnmj6

Address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Datacap Allocated

256.00TiB

Signer Address

f1e77zuityhvvw6u2t6tb5qlnsegy2s67qs4lbbbq

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebirygsqvde5blaz356gclscvjih52vgis5qjkj5upqicvagcnmj6

NDLABS-Leo commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacediszg4l765ffemh75grtb4lounygiehgguclxafokxe7uqa3p3hk

Address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Datacap Allocated

256.00TiB

Signer Address

f1yayfsv6whu3rheviucvventj3y6t542xfpb47ei

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacediszg4l765ffemh75grtb4lounygiehgguclxafokxe7uqa3p3hk

large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!
large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f01858410

Client address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

DataCap allocation requested

512TiB

Id

5a789d22-e115-45c5-9315-350ec0702f37

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Last two approvers

not found & newwebgroup

Rule to calculate the allocation request amount

10% of total dc amount requested

DataCap allocation requested

512TiB

Total DataCap granted for client so far

256TiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

4.75PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
6225 10 256TiB 25.35 29.93TiB
ipfscn commented 1 year ago

looks good

ipfscn commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebbwiw2zc2cjfadws6smmhvklrtv2vhn4e4qjhiifkxnqhujoldzq

Address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Datacap Allocated

512.00TiB

Signer Address

f1j4n74chme7whbz3yls4a7ixqewb6dijypqg2a3a

Id

5a789d22-e115-45c5-9315-350ec0702f37

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebbwiw2zc2cjfadws6smmhvklrtv2vhn4e4qjhiifkxnqhujoldzq

xinaxu commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecosgyi265lwi5o6bgybzkjges4kydsyielkadiog4qkqojejpw2k

Address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Datacap Allocated

512.00TiB

Signer Address

f1k3ysofkrrmqcot6fkx4wnezpczlltpirmrpsgui

Id

5a789d22-e115-45c5-9315-350ec0702f37

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecosgyi265lwi5o6bgybzkjges4kydsyielkadiog4qkqojejpw2k

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f01858410

Client address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

DataCap allocation requested

1PiB

Id

c0653e74-9e21-4875-9fd1-3fff7a3a28d8

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Last two approvers

xinaxu & ipfscn

Rule to calculate the allocation request amount

20% of total dc amount requested

DataCap allocation requested

1PiB

Total DataCap granted for client so far

256TiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

4.75PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
8068 10 512TiB 19.56 2.09TiB
1ane-1 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceailcmyyo6doksael42aartvzpt2fd2pmkwrs6f7qub5zfpf5xiqe

Address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Datacap Allocated

1.00PiB

Signer Address

f1mdk7s2vntzm6hu35yuo6vjubtrpfnb2awhgvrri

Id

c0653e74-9e21-4875-9fd1-3fff7a3a28d8

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceailcmyyo6doksael42aartvzpt2fd2pmkwrs6f7qub5zfpf5xiqe

cryptowhizzard commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaced37kd4ziwgfrupqlgb3r3nqyzaxduadmygr7q6lfe2rmwkwlermi

Address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Datacap Allocated

1.00PiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

c0653e74-9e21-4875-9fd1-3fff7a3a28d8

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaced37kd4ziwgfrupqlgb3r3nqyzaxduadmygr7q6lfe2rmwkwlermi

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f01858410

Client address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

DataCap allocation requested

2PiB

Id

21705f77-e70c-40d5-8d35-4a9bab7eca1a

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Last two approvers

cryptowhizzard & 1ane-1

Rule to calculate the allocation request amount

40% of total dc amount requested

DataCap allocation requested

2PiB

Total DataCap granted for client so far

1.75PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

3.25PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
50131 13 1PiB 13.81 164.43TiB
liyunzhi-666 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceahmr55s5fyd3qf6ixp7r3ensvqhtiztriomcdxu6xg3dnyqqbqj4

Address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Datacap Allocated

2.00PiB

Signer Address

f1pszcrsciyixyuxxukkvtazcokexbn54amf7gvoq

Id

21705f77-e70c-40d5-8d35-4a9bab7eca1a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceahmr55s5fyd3qf6ixp7r3ensvqhtiztriomcdxu6xg3dnyqqbqj4

NiwanDao commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedswlv45nis6s25ygwe4d6lbvvd2zwgd2mqxvmcklrolfj53sqlls

Address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Datacap Allocated

2.00PiB

Signer Address

f1a2lia2cwwekeubwo4nppt4v4vebxs2frozarz3q

Id

21705f77-e70c-40d5-8d35-4a9bab7eca1a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedswlv45nis6s25ygwe4d6lbvvd2zwgd2mqxvmcklrolfj53sqlls

NiwanDao commented 1 year ago

Deal distribution is decentralized for now. Approved.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f01858410

Client address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

DataCap allocation requested

1.25PiB

Id

7219468c-bf36-4a27-8452-15e0eb989082

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Last two approvers

xingjitansuo & liyunzhi-666

Rule to calculate the allocation request amount

80% of total dc amount requested

DataCap allocation requested

1.25PiB

Total DataCap granted for client so far

1.75PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

3.25PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
50131 13 2PiB 13.81 164.43TiB
filplus-checker commented 1 year ago

DataCap and CID Checker Report[^1]

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

✔️ Storage provider distribution looks healthy.

Provider Location Total Deals Sealed Percentage Unique Data Duplicate Deals
f01946713 Singapore, Singapore, SG 297.13 TiB 13.88% 297.13 TiB 0.00%
f01945216 Singapore, Singapore, SG 295.97 TiB 13.82% 295.97 TiB 0.00%
f01947708 Tokyo, Tokyo, JP 293.50 TiB 13.71% 293.50 TiB 0.00%
f01964074 Tokyo, Tokyo, JP 291.72 TiB 13.62% 291.72 TiB 0.00%
f01963842 Tokyo, Tokyo, JP 284.41 TiB 13.28% 284.41 TiB 0.00%
f01946689 Singapore, Singapore, SG 281.63 TiB 13.15% 281.63 TiB 0.00%
f01944744 Hong Kong, Central and Western, HK 254.91 TiB 11.90% 254.91 TiB 0.00%
f01928097 Hong Kong, Central and Western, HK 49.22 TiB 2.30% 49.22 TiB 0.00%
f01918046 Kuala Lumpur, Kuala Lumpur, MY 18.75 TiB 0.88% 18.75 TiB 0.00%
f01918045 Kuala Lumpur, Kuala Lumpur, MY 18.75 TiB 0.88% 18.75 TiB 0.00%
f01909705 Kuala Lumpur, Kuala Lumpur, MY 18.75 TiB 0.88% 18.75 TiB 0.00%
f01888767 West Jordan, Utah, US 18.38 TiB 0.86% 18.38 TiB 0.00%
f01986353new Shanghai, Shanghai, CN 18.34 TiB 0.86% 18.34 TiB 0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

✔️ Data replication looks healthy.

Unique Data Size Total Deals Made Number of Providers Deal Percentage
38.19 TiB 38.19 TiB 1 1.78%
3.00 TiB 6.00 TiB 2 0.28%
21.03 TiB 63.09 TiB 3 2.95%
7.03 TiB 28.13 TiB 4 1.31%
4.38 TiB 21.88 TiB 5 1.02%
27.88 TiB 167.25 TiB 6 7.81%
204.34 TiB 1.40 PiB 7 66.80%
48.31 TiB 386.50 TiB 8 18.05%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client Application Total Deals Affected Unique CIDs Verifier
f1itrbkc7i4u2p46pb6ugv3bmz6zmdjvelpmf6vqi Fei Yan - Kernelogic 8.38 TiB 268 LDN v3 multisig

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

cryptowhizzard commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecxzlkh4hmntjdfsdzsnx5adntpnlua2thgppuh5hotpnuo63izfm

Address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Datacap Allocated

1.25PiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

7219468c-bf36-4a27-8452-15e0eb989082

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecxzlkh4hmntjdfsdzsnx5adntpnlua2thgppuh5hotpnuo63izfm

newwebgroup commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebmumhimzhtv2hgdkownvajypqrss256wlisttt2ffqr63vhth2es

Address

f1ua4eolclcnc2pzp4tsqodey3rwbjz46slb6e3nq

Datacap Allocated

1.25PiB

Signer Address

f1e77zuityhvvw6u2t6tb5qlnsegy2s67qs4lbbbq

Id

7219468c-bf36-4a27-8452-15e0eb989082

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebmumhimzhtv2hgdkownvajypqrss256wlisttt2ffqr63vhth2es

kernelogic commented 1 year ago

Fully allocated. Closing issue. Thanks everyone!

large-datacap-requests[bot] commented 3 months ago

Thanks for your request! :exclamation: We have found some problems in the information provided. We could not find Organization Name field in the information provided We could not find Website \/ Social Media field in the information provided We could not find Total amount of DataCap being requested (between 500 TiB and 5 PiB) field in the information provided We could not find Weekly allocation of DataCap requested (usually between 1-100TiB) field in the information provided We could not find On-chain address for first allocation field in the information provided We could not find Data Type of Application field in the information provided

Please, take a look at the request and edit the body of the issue providing all the required information.