filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
109 stars 62 forks source link

[DataCap Application] Twin Quasar - Nasa Nex #2217

Open s0nik42 opened 1 year ago

s0nik42 commented 1 year ago

Data Owner Name

Nasa

What is your role related to the dataset

Data onramp entity that provides data onboarding services to multiple clients

Data Owner Country/Region

France

Data Owner Industry

Life Science / Healthcare

Website

https://twinquasar.io

Social Media

https://www.linkedin.com/company/79602274
https://www.cidgravity.com/
https://twitter.com/cidgravity
https://www.plnetwork.io/directory/members/cldvoqbz00axru21klqzlosgx

Total amount of DataCap being requested

1PiB

Expected size of single dataset (one copy)

200TiB

Number of replicas to store

5

Weekly allocation of DataCap requested

100TiB

On-chain address for first allocation

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

Data Type of Application

Slingshot

Custom multisig

Identifier

No response

Share a brief history of your project and organization

At twinquasar we are onboarding data to Slingshot since Slingshot 2.2 . Our goal is to onboard useful data to the network in a reusable way.

Twinquasar has many roles in the ecosystem :
- Filecoin expertise : consultancy / managed services
- SP operations
- Filecoin notary
- Developper of CIDgravity
- Developper of Lotus Farcaster
- Environmental web3 developer

Is this project associated with other projects/ecosystem stakeholders?

Yes

If answered yes, what are the other projects/ecosystem stakeholders

PL via the Slinshot challenge

Describe the data being stored onto Filecoin

NASA NEX dataset

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

Nasa NEX is a collection of Earth science datasets maintained by NASA, including climate change projections and satellite images of the Earth's surface.

The dataset is coming from AWS public S3 repo  : https://registry.opendata.aws/nasanex/

If you are a data preparer. What is your location (Country/Region)

France

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

- Singularity
- motion (testing)
- CIDgravity
- boost

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

The distribution will be world wide, the data will be proposed to 100 SPs using CIDgravity; the distribution will be automated and based on SP availability and success ratio.

< 3% of that dataset will reach our own SPs.

We have no business interest in the other SPs we are going to store to, the distribution will be purely based on performance criterias.  

We take the opportunity of onboarding this dataset to see how CIDgravity can improve the data success rate of the "Filecoin RES API" stack

Please share a sample of the data

A link to 

Available at s3://nasanex/NEX-DCP30

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Daily

For how long do you plan to keep this dataset stored on Filecoin

More than 3 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, Africa, North America, South America, Europe, Australia (continent), Antarctica

How will you be distributing your data to storage providers

Lotus built-in data transfer

How do you plan to choose storage providers

Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

Selection process is explained in a section above

If you already have a list of storage providers to work with, fill out their names and provider IDs below

No response

How do you plan to make deals to your storage providers

Boost client, Singularity

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

data-programs commented 1 year ago
KYC

This user’s identity has been verified through filplus.storage

herrehesse commented 1 year ago

Highly intrigued by the worldwide distribution capabilities of this dataset through CIDgravity. This approach greatly surpasses the typical (abusive) methods involving 10-15PiB throwaway accounts that aim to exploit the system.

Thanks to @s0nik42 for the LDN request.

s0nik42 commented 1 year ago

@ze42

Sunnyiscoming commented 1 year ago
jamerduhgamer commented 1 year ago

Willing to support CID Gravity and their efforts to propose deals through their application. Data distribution will be worldwide as well which is good.

s0nik42 commented 1 year ago

@Sunnyiscoming if you read the application, the idea is to spread that dataset across random SPs accross the globe. I cannot provide contact information at these SP entities. They are not even aware except reading thiese lines they will receive deals... let me know how we can move on before the dev summit in iceland.

s0nik42 commented 1 year ago

I submitted the form please review and let me know if we can proceed with the allocation. Thanks

dannyob commented 1 year ago

I don't see this appearing on the https://filplus.fil.org/ app -- @kevzak @galen-mcandrew is there anything that needs to be done?

simonkim0515 commented 1 year ago

Datacap Request Trigger

Total DataCap requested

1PiB

Expected weekly DataCap usage rate

200TiB

Client address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

DataCap allocation requested

50TiB

Id

921d7866-c9df-4b59-82b6-24e1c129d613

kevzak commented 1 year ago

I don't see this appearing on the https://filplus.fil.org/ app -- @kevzak @galen-mcandrew is there anything that needs to be done?

@dannyob it doesn't show until after the trigger. It should be there now

cryptowhizzard commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedhab3jz3fgre7hhhqljqr4nndq4uw6f6odun3liphr3e4uf353fs

Address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

Datacap Allocated

50.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

921d7866-c9df-4b59-82b6-24e1c129d613

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedhab3jz3fgre7hhhqljqr4nndq4uw6f6odun3liphr3e4uf353fs

jamerduhgamer commented 1 year ago

Willing to sign the first 50 TiB datacap tranche and see how the project launches!

jamerduhgamer commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebnjuvqx6qyatuavxpnxbhzsrndvqwbdc7h3fycug75h6zwbmokne

Address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

Datacap Allocated

50.00TiB

Signer Address

f1kqdiokoeubyse4qpihf7yrpl7czx4qgupx3eyzi

Id

921d7866-c9df-4b59-82b6-24e1c129d613

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebnjuvqx6qyatuavxpnxbhzsrndvqwbdc7h3fycug75h6zwbmokne

s0nik42 commented 1 year ago

Hey, @kevzak @Sunnyiscoming

can you trigger the next tranch please ?

thank you

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f01858389

Client address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

DataCap allocation requested

100TiB

Id

580a9bfd-9b68-44fd-a509-2ee5ae721002

cryptowhizzard commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceajeue36cytjotg7dlxtbv3gmwga5lfzpf3v5lfazus7j7m4k6fdk

Address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

Datacap Allocated

100.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

580a9bfd-9b68-44fd-a509-2ee5ae721002

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceajeue36cytjotg7dlxtbv3gmwga5lfzpf3v5lfazus7j7m4k6fdk

GaryGJG commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

liyunzhi-666 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

BlockMakeronline commented 1 year ago

The data supports retrieval and the storage looks good.

PluskitOfficial commented 1 year ago

Loolks good. Willing to support.

Fenbushi-Filecoin commented 1 year ago

Love to support!

Fenbushi-Filecoin commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacec4vp2y3qa6ldt2du74owwjqsjxgg3b2nvx57kgfgf6jdinxjm6ow

Address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

Datacap Allocated

100.00TiB

Signer Address

f1yqydpmqb5en262jpottko2kd65msajax7fi4rmq

Id

580a9bfd-9b68-44fd-a509-2ee5ae721002

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec4vp2y3qa6ldt2du74owwjqsjxgg3b2nvx57kgfgf6jdinxjm6ow

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

DataCap allocation requested

100TiB

Id

c58a7ed1-6108-4c53-8b7f-320bb4712d71

cryptowhizzard commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebgwoouvm7s5acnuf37ce5i7ylpcbj5j2cbaale425nr5rq5hdirw

Address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

Datacap Allocated

100.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

c58a7ed1-6108-4c53-8b7f-320bb4712d71

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebgwoouvm7s5acnuf37ce5i7ylpcbj5j2cbaale425nr5rq5hdirw

s0nik42 commented 1 year ago

@Sunnyiscoming @Kevin-FF-USA @simonkim0515 Guys, there is something wrong here : everytime a notary sign it, the system just proposed it again endlessly by replacing the previous proposal and it's never going futher

MatrixStorage commented 1 year ago

I am willing to support and will try to approve

MatrixStorage commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebhq2wwlc6ojvuhkxpxnu7kijvhi2evu6b3fx7dis4shdfco3vgb2

Address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

Datacap Allocated

100.00TiB

Signer Address

f1tbxqwjxfyv7swsdin4einirlsfquv3vnmlapley

Id

c58a7ed1-6108-4c53-8b7f-320bb4712d71

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebhq2wwlc6ojvuhkxpxnu7kijvhi2evu6b3fx7dis4shdfco3vgb2

MatrixStorage commented 1 year ago

I have finished the approval @s0nik42

s0nik42 commented 1 year ago

Thank you @MatrixStorage !

kevzak commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 99.76% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard.

kevzak commented 1 year ago

watch your distribution numbers @s0nik42 thanks

herrehesse commented 1 year ago
Screenshot 2023-11-02 at 15 49 16

Your regional spread looks amazing and ZERO VPN. Wish more applications looked like this @s0nik42 .

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

DataCap allocation requested

200TiB

Id

7e118a3d-c2cf-4255-a416-421cd26c3f4a

ghost commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard.

herrehesse commented 1 year ago

Screenshot 2023-11-20 at 16 16 12

@s0nik42 extremely healthy distribution. Thank you.

s0nik42 commented 1 year ago

yes thanks you !

jamerduhgamer commented 1 year ago

Application looks good. Willing to continue supporting.

jamerduhgamer commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceadbmasy7rputi7fjhcik5uc6bfhw7rcrselvpmuhqcbwziszxcb2

Address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

Datacap Allocated

200.00TiB

Signer Address

f1kqdiokoeubyse4qpihf7yrpl7czx4qgupx3eyzi

Id

7e118a3d-c2cf-4255-a416-421cd26c3f4a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceadbmasy7rputi7fjhcik5uc6bfhw7rcrselvpmuhqcbwziszxcb2

s0nik42 commented 1 year ago

Hi, @Sunnyiscoming @galen-mcandrew @Kevin-FF-USA @simonkim0515 is there any remaining datacap on the LDN wallet ?

cryptowhizzard commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebfk27xwlpw3lfybmlwwcsbebgd3nlptrmmjnfwy6pu6gfjrups6u

Address

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i

Datacap Allocated

200.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

7e118a3d-c2cf-4255-a416-421cd26c3f4a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebfk27xwlpw3lfybmlwwcsbebgd3nlptrmmjnfwy6pu6gfjrups6u

github-actions[bot] commented 11 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

github-actions[bot] commented 11 months ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

-- Commented by Stale Bot.

s0nik42 commented 10 months ago

checker:manualTrigger