search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
109 stars 62 forks source link

[DataCap Application] <Outercore> - <Throughput> #2106

Closed corinne-antonia closed 1 year ago

corinne-antonia commented 1 year ago

Data Owner Name

Outercore - Network Growth - Engineering

What is your role related to the dataset

Data onramp entity that provides data onboarding services to multiple clients

Data Owner Country/Region

United States

Data Owner Industry

Web3 / Crypto

Website

https://fw.services

Social Media

https://twitter.com/OutercoreEng
https://twitter.com/Estuary_Tech

Total amount of DataCap being requested

1PiB

Expected size of single dataset (one copy)

166TiB

Number of replicas to store

6

Weekly allocation of DataCap requested

10TiB

On-chain address for first allocation

f1xhiluhol36nj6t33lekpc5egjtusc3d7kbqeh2i

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Identifier

No response

Share a brief history of your project and organization

Root website: https://fw.services
Project website: https://delta.store

We are the Engineering team at Protocol Labs ➝ Outercore ➝ Engineering/Network Growth. Our goal is develop tooling for entire Filecoin Ecosystem to use to onboard data to the Filecoin Network.

This is the fourth application for our tool Delta. The first one is located here. 
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1854
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/160. The second one is here: https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1602

#### Our mission

We want an internet where resources are owned & shared by everyone. Everyone shares consensus over a distributed ledger. Data storage is verifiably stored for the end-user, and more fault tolerant. There is no central authority of control and no single point of failure. And the security promise only improves as the network continues to grow. Just imagine all of the certifications that can be automatically generated for end users by a new network like this. This is how you achieve storage as a human right for anyone in the world.

We're going to prove that we can get here by building the tools that can support onboarding 10 PiBs a day of data to the Filecoin Network. But we're also not going to forget about retrievability and helping the Filecoin Network actually become usable.

#### Delta

Our solution to archival and cold storage use cases.

Use ∆ Delta to upload all of your useful public data to Filecoin storage providers. Delta is a straight-forward Filecoin storage deal making tool that manages deals, and does not do anything else. It is purely for the function of helping Storage Providers fill capacity either through online or offline methods. It is written in Go and designed to be paired well with bare-metal infrastructure.

Is this project associated with other projects/ecosystem stakeholders?

Yes

If answered yes, what are the other projects/ecosystem stakeholders

Protocol Labs -> Outercore -> Network Growth

Describe the data being stored onto Filecoin

We are working directly with Throughput (https://www.throughput.network/). They have given us permission to upload their data to Filecoin on their behalf using our tooling, Delta. 

Throughput is a blockchain that offers "ledger as a service." Throughput employs a single process that maintains multiple ledgers belonging to different blockchains. This approach provides a secure and efficient means of maintaining multiple ledgers, reducing the complexity and cost of running a layer 1 blockchain.

The types of data that will be onboarded are Datasets (CSV, SSL certificates, Large AI/ML data, Image datasets, Blockchain Ledger).

Where was the data currently stored in this dataset sourced from

Other

If you answered "Other" in the previous question, enter the details here

S3, GCP, Dropbox

How do you plan to prepare the dataset

No response

If you answered "other/custom tool" in the previous question, enter the details here

Our new tool Delta

Learn more at https://delta.store

Please share a sample of the data

Some examples below:

Defi applications to store user metadata (encrypted)
Blockchain infrastructure metadata and backups
Crypto wallet metadata (encrypted)
Machine Learning data sets
Blockchain transactions 

[[{"id":0,"hash":"0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43","previous_hash":"genesis","timestamp":1662048082,"data":"genesis!","nonce":2836},{"id":1,"hash":"000029de9f887ebbaa596f688f46490a59b6f743c403b3c8d8ed17e7cf7a9be5","previous_hash":"0000f816a87f806bb0073dcf026a64fb40c946b5abee2573702828694d5b4c43","timestamp":1662048123,"data":" $alksdjflkasjdfjasdjf","nonce":15628},{"id":2,"hash":"00006d8d89609a4e0738a7f6858a473b2d58eaceb47ac2cd7c3099eb5b770f7e","previous_hash":"000029de9f887ebbaa596f688f46490a59b6f743c403b3c8d8ed17e7cf7a9be5","timestamp":1662048153,"data":" $ajsdfljasldfjjasldfjsajdfj","nonce":56811},{"id":3,"hash":"00008ea28973b8ddb50e4b666367a4fe6b387322d7bae37b1acd94d230916a02","previous_hash":"00006d8d89609a4e0738a7f6858a473b2d58eaceb47ac2cd7c3099eb5b770f7e","timestamp":1662050043,"data":" $datafromnode3","nonce":52567},{"id":4,"hash":"00004b7bdb3a8dd3b2e72b72a5068d81c220eb5ba60ac226103c97314966b150","previous_hash":"00008ea28973b8ddb50e4b666367a4fe6b387322d7bae37b1acd94d230916a02","timestamp":1662050214,"data":"create b $datafromnode3ccreate b $datafrocreate b create b $datafromnode3","nonce":4061},{"id":5,"hash":"00001bf782c42515668e9a117ebcbab401411f070bf2e6183e67daf771503bfe","previous_hash":"00004b7bdb3a8dd3b2e72b72a5068d81c220eb5ba60ac226103c97314966b150","timestamp":1662050482,"data":" $datafromcreate b $datafromnode2","nonce":10242},{"id":6,"hash":"00001b109a621c0c832e84102b61d22573f05892d009234c594a15d55c7ae91d","previous_hash":"00001bf782c42515668e9a117ebcbab401411f070bf2e6183e67daf771503bfe","timestamp":1662051839,"data":" $datafromone1","nonce":121946}]]


### Confirm that this is a public dataset that can be retrieved by anyone on the Network

- [X] I confirm

### If you chose not to confirm, what was the reason

```text
Delta is not focused on retrieval, its focused on storage onboarding. We are going to help all SPs onboard all of their data.

Estuary is focused on retrieval.

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

More than 3 years

In which geographies do you plan on making storage deals

North America

How will you be distributing your data to storage providers

Others

How do you plan to choose storage providers

Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

We will be developing our tool Delta

Learn more at https://delta.store

If you already have a list of storage providers to work with, fill out their names and provider IDs below

We have developed our own list at https://data.storage.market but we will work with everyone/anyone. We actively work with storage providers and we have a storage provider on our team.

93 of them that we plan to start with, obviously we'll work with more.

➝ https://data.storage.market/api/providers/f0840770
➝ https://data.storage.market/api/providers/f01624021
➝ https://data.storage.market/api/providers/f01806491
➝ https://data.storage.market/api/providers/f08399
➝ https://data.storage.market/api/providers/f01790264
➝ https://data.storage.market/api/providers/f0875769
➝ https://data.storage.market/api/providers/f01035680
➝ https://data.storage.market/api/providers/f033356
➝ https://data.storage.market/api/providers/f01683871
➝ https://data.storage.market/api/providers/f03488
➝ https://data.storage.market/api/providers/f030379
➝ https://data.storage.market/api/providers/f01466075
➝ https://data.storage.market/api/providers/f02301
➝ https://data.storage.market/api/providers/f010479
➝ https://data.storage.market/api/providers/f010088
➝ https://data.storage.market/api/providers/f0773157
➝ https://data.storage.market/api/providers/f0717969
➝ https://data.storage.market/api/providers/f0461791
➝ https://data.storage.market/api/providers/f01746964
➝ https://data.storage.market/api/providers/f01059489
➝ https://data.storage.market/api/providers/f023467
➝ https://data.storage.market/api/providers/f01392893
➝ https://data.storage.market/api/providers/f01736668
➝ https://data.storage.market/api/providers/f022352
➝ https://data.storage.market/api/providers/f01199430
➝ https://data.storage.market/api/providers/f09848
➝ https://data.storage.market/api/providers/f01222595
➝ https://data.storage.market/api/providers/f01794610
➝ https://data.storage.market/api/providers/f01443744
➝ https://data.storage.market/api/providers/f01199442
➝ https://data.storage.market/api/providers/f02401
➝ https://data.storage.market/api/providers/f01402814
➝ https://data.storage.market/api/providers/f0104671
➝ https://data.storage.market/api/providers/f01207045
➝ https://data.storage.market/api/providers/f0406703
➝ https://data.storage.market/api/providers/f01175097
➝ https://data.storage.market/api/providers/f010446
➝ https://data.storage.market/api/providers/f0724219
➝ https://data.storage.market/api/providers/f022142
➝ https://data.storage.market/api/providers/f058369
➝ https://data.storage.market/api/providers/f01201327
➝ https://data.storage.market/api/providers/f0406322
➝ https://data.storage.market/api/providers/f01278
➝ https://data.storage.market/api/providers/f0187709
➝ https://data.storage.market/api/providers/f01045784
➝ https://data.storage.market/api/providers/f0706693
➝ https://data.storage.market/api/providers/f024184
➝ https://data.storage.market/api/providers/f01385207
➝ https://data.storage.market/api/providers/f01652333
➝ https://data.storage.market/api/providers/f01319368
➝ https://data.storage.market/api/providers/f0214334
➝ https://data.storage.market/api/providers/f082635
➝ https://data.storage.market/api/providers/f0836160
➝ https://data.storage.market/api/providers/f0135078
➝ https://data.storage.market/api/providers/f039940
➝ https://data.storage.market/api/providers/f0408717
➝ https://data.storage.market/api/providers/f01345523
➝ https://data.storage.market/api/providers/f01108096
➝ https://data.storage.market/api/providers/f01611097
➝ https://data.storage.market/api/providers/f01208862
➝ https://data.storage.market/api/providers/f01662356
➝ https://data.storage.market/api/providers/f01423116
➝ https://data.storage.market/api/providers/f017665
➝ https://data.storage.market/api/providers/f01666984
➝ https://data.storage.market/api/providers/f0440429
➝ https://data.storage.market/api/providers/f01028552
➝ https://data.storage.market/api/providers/f0707721
➝ https://data.storage.market/api/providers/f099608
➝ https://data.storage.market/api/providers/f0142637
➝ https://data.storage.market/api/providers/f01127678
➝ https://data.storage.market/api/providers/f0501283
➝ https://data.storage.market/api/providers/f01133080
➝ https://data.storage.market/api/providers/f0127896
➝ https://data.storage.market/api/providers/f010617
➝ https://data.storage.market/api/providers/f097777
➝ https://data.storage.market/api/providers/f01367109
➝ https://data.storage.market/api/providers/f01225882
➝ https://data.storage.market/api/providers/f023971
➝ https://data.storage.market/api/providers/f0466405
➝ https://data.storage.market/api/providers/f01310564
➝ https://data.storage.market/api/providers/f02620
➝ https://data.storage.market/api/providers/f02576
➝ https://data.storage.market/api/providers/f01764587
➝ https://data.storage.market/api/providers/f019551
➝ https://data.storage.market/api/providers/f034258
➝ https://data.storage.market/api/providers/f0240185
➝ https://data.storage.market/api/providers/f01619524
➝ https://data.storage.market/api/providers/f01096124
➝ https://data.storage.market/api/providers/f01163272
➝ https://data.storage.market/api/providers/f01337533
➝ https://data.storage.market/api/providers/f01091851
➝ https://data.storage.market/api/providers/f01784458
➝ https://data.storage.market/api/providers/f08403

How do you plan to make deals to your storage providers

No response

If you answered "Others/custom tool" in the previous question, enter the details here

Our new tool Delta, which we plan to release to the whole ecosystem.

Learn more at https://delta.store

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

kevzak commented 1 year ago

Data Preparation clear. List of Delta onboarding tool vetted SPs being used clear. @Sunnyiscoming please trigger this application.

Sunnyiscoming commented 1 year ago

Datacap Request Trigger

Total DataCap requested

1PiB

Expected weekly DataCap usage rate

10TiB

Client address

f1xhiluhol36nj6t33lekpc5egjtusc3d7kbqeh2i

cryptowhizzard commented 1 year ago

@corinne-antonia requested that I endorse both outercore applications. Given their affiliation with PL & Estuary, their reputation is exceptionally strong. Observing their client and data onboarding strategies, it becomes evident that they are an ideal candidate for Filecoin+ support.

They have our complete backing. We eagerly anticipate receiving their CID & Retrieval reports.

kevzak commented 1 year ago

Datacap Request Trigger

Total DataCap requested

1PiB

Expected weekly DataCap usage rate

10TiB

Client address

f1xhiluhol36nj6t33lekpc5egjtusc3d7kbqeh2i

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1xhiluhol36nj6t33lekpc5egjtusc3d7kbqeh2i

DataCap allocation requested

5TiB

Id

4ca77459-327a-4748-833c-88a7121f5790

kevzak commented 1 year ago

@corinne-antonia I had to unmark "Custom Multisig" question. In the future, do not mark that when applying.

This application should be available to sign FYI @cryptowhizzard . CC @simonkim0515

cryptowhizzard commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacea5c7u4mdtzdrshw4xazjmzbfd6jgp7b4tfe6p5af3cbalfipoakg

Address

f1xhiluhol36nj6t33lekpc5egjtusc3d7kbqeh2i

Datacap Allocated

5.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

4ca77459-327a-4748-833c-88a7121f5790

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea5c7u4mdtzdrshw4xazjmzbfd6jgp7b4tfe6p5af3cbalfipoakg

liyunzhi-666 commented 1 year ago

Hi @corinne-antonia,as far as I know, the Outercore team has been disbanded. Is this LDN application still open?

MEIYAN666 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedwasdbogabogomy666a2xyfpqivuvg7wc2nazyzotoxnveip2g3e

Address

f1xhiluhol36nj6t33lekpc5egjtusc3d7kbqeh2i

Datacap Allocated

5.00TiB

Signer Address

f1bwugfihrmn3iyunzyxst5nttql3dge4khwmurtq

Id

4ca77459-327a-4748-833c-88a7121f5790

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedwasdbogabogomy666a2xyfpqivuvg7wc2nazyzotoxnveip2g3e

MEIYAN666 commented 1 year ago

LGTM, willing to support!

corinne-antonia commented 1 year ago

Hi @corinne-antonia,as far as I know, the Outercore team has been disbanded. Is this LDN application still open?

Yes these are still open. I will still be opening LDNs under "Outercore" as we still have these client relationships within the network growth team.

liyunzhi-666 commented 1 year ago

Hi @corinne-antonia,as far as I know, the Outercore team has been disbanded. Is this LDN application still open?

Yes these are still open. I will still be opening LDNs under "Outercore" as we still have these client relationships within the network growth team.

It was supported by others in this round, and you can continue to ask me to support your application in the next round.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!