filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
109 stars 62 forks source link

[DataCap Application] <Outercore> - <Decentr.ai> #2133

Closed corinne-antonia closed 1 year ago

corinne-antonia commented 1 year ago

Data Owner Name

Outercore - Network Growth - Engineering

What is your role related to the dataset

Data onramp entity that provides data onboarding services to multiple clients

Data Owner Country/Region

United States

Data Owner Industry

Web3 / Crypto

Website

https://fw.services

Social Media

https://twitter.com/OutercoreEng
https://twitter.com/Estuary_Tech
https://twitter.com/FilecoinTools

Total amount of DataCap being requested

1PiB

Expected size of single dataset (one copy)

166TiB

Number of replicas to store

6

Weekly allocation of DataCap requested

10TiB

On-chain address for first allocation

f1myzxkfsadel3n4liaw6y53s525txm7fvsia4m6q

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Identifier

No response

Share a brief history of your project and organization

Root website: https://fw.services
Project website: https://delta.store

We are the Engineering team at Protocol Labs ➝ Outercore ➝ Engineering/Network Growth. Our goal is develop tooling for entire Filecoin Ecosystem to use to onboard data to the Filecoin Network.

This is the fourth application for our tool Delta. The first one is located here. 
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1854
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/160. The second one is here: https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1602

#### Our mission

We want an internet where resources are owned & shared by everyone. Everyone shares consensus over a distributed ledger. Data storage is verifiably stored for the end-user, and more fault tolerant. There is no central authority of control and no single point of failure. And the security promise only improves as the network continues to grow. Just imagine all of the certifications that can be automatically generated for end users by a new network like this. This is how you achieve storage as a human right for anyone in the world.

We're going to prove that we can get here by building the tools that can support onboarding 10 PiBs a day of data to the Filecoin Network. But we're also not going to forget about retrievability and helping the Filecoin Network actually become usable.

#### Delta

Our solution to archival and cold storage use cases.

Use ∆ Delta to upload all of your useful public data to Filecoin storage providers. Delta is a straight-forward Filecoin storage deal making tool that manages deals, and does not do anything else. It is purely for the function of helping Storage Providers fill capacity either through online or offline methods. It is written in Go and designed to be paired well with bare-metal infrastructure.

Is this project associated with other projects/ecosystem stakeholders?

Yes

If answered yes, what are the other projects/ecosystem stakeholders

Protocol Labs -> Outercore -> Network Growth

Describe the data being stored onto Filecoin

We are working directly with Decentr.ai (https://decentr.ai/). They have given us permission to upload their data to Filecoin on their behalf using our tooling, Delta. 

- Machine Learning Datasets
- Unstructured datasets (for modelling)
- Resulting models for AI

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

S3

If you are a data preparer. What is your location (City and Country)

(Dallas) United States

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

Our new tool Delta

Learn more at https://delta.store

If you are not preparing the data, who will prepare the data? (Provide name and business)

We are preparing the data

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

No

Please share a sample of the data

https://rdm.uq.edu.au/files/c31a9f50-ef99-11ed-ab7b-c7846b13c8a9

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

Delta is not focused on retrieval, its focused on storage onboarding. We are going to help all SPs onboard all of their data.

Estuary is focused on retrieval.

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

More than 3 years

In which geographies do you plan on making storage deals

North America

How will you be distributing your data to storage providers

Others

How do you plan to choose storage providers

Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

We will be developing our tool Delta

Learn more at https://delta.store

If you already have a list of storage providers to work with, fill out their names and provider IDs below

We have developed our own list at https://data.storage.market but we will work with everyone/anyone. We actively work with storage providers and we have a storage provider on our team.

93 of them that we plan to start with, obviously we'll work with more.

➝ https://data.storage.market/api/providers/f0840770
➝ https://data.storage.market/api/providers/f01624021
➝ https://data.storage.market/api/providers/f01806491
➝ https://data.storage.market/api/providers/f08399
➝ https://data.storage.market/api/providers/f01790264
➝ https://data.storage.market/api/providers/f0875769
➝ https://data.storage.market/api/providers/f01035680
➝ https://data.storage.market/api/providers/f033356
➝ https://data.storage.market/api/providers/f01683871
➝ https://data.storage.market/api/providers/f03488
➝ https://data.storage.market/api/providers/f030379
➝ https://data.storage.market/api/providers/f01466075
➝ https://data.storage.market/api/providers/f02301
➝ https://data.storage.market/api/providers/f010479
➝ https://data.storage.market/api/providers/f010088
➝ https://data.storage.market/api/providers/f0773157
➝ https://data.storage.market/api/providers/f0717969
➝ https://data.storage.market/api/providers/f0461791
➝ https://data.storage.market/api/providers/f01746964
➝ https://data.storage.market/api/providers/f01059489
➝ https://data.storage.market/api/providers/f023467
➝ https://data.storage.market/api/providers/f01392893
➝ https://data.storage.market/api/providers/f01736668
➝ https://data.storage.market/api/providers/f022352
➝ https://data.storage.market/api/providers/f01199430
➝ https://data.storage.market/api/providers/f09848
➝ https://data.storage.market/api/providers/f01222595
➝ https://data.storage.market/api/providers/f01794610
➝ https://data.storage.market/api/providers/f01443744
➝ https://data.storage.market/api/providers/f01199442
➝ https://data.storage.market/api/providers/f02401
➝ https://data.storage.market/api/providers/f01402814
➝ https://data.storage.market/api/providers/f0104671
➝ https://data.storage.market/api/providers/f01207045
➝ https://data.storage.market/api/providers/f0406703
➝ https://data.storage.market/api/providers/f01175097
➝ https://data.storage.market/api/providers/f010446
➝ https://data.storage.market/api/providers/f0724219
➝ https://data.storage.market/api/providers/f022142
➝ https://data.storage.market/api/providers/f058369
➝ https://data.storage.market/api/providers/f01201327
➝ https://data.storage.market/api/providers/f0406322
➝ https://data.storage.market/api/providers/f01278
➝ https://data.storage.market/api/providers/f0187709
➝ https://data.storage.market/api/providers/f01045784
➝ https://data.storage.market/api/providers/f0706693
➝ https://data.storage.market/api/providers/f024184
➝ https://data.storage.market/api/providers/f01385207
➝ https://data.storage.market/api/providers/f01652333
➝ https://data.storage.market/api/providers/f01319368
➝ https://data.storage.market/api/providers/f0214334
➝ https://data.storage.market/api/providers/f082635
➝ https://data.storage.market/api/providers/f0836160
➝ https://data.storage.market/api/providers/f0135078
➝ https://data.storage.market/api/providers/f039940
➝ https://data.storage.market/api/providers/f0408717
➝ https://data.storage.market/api/providers/f01345523
➝ https://data.storage.market/api/providers/f01108096
➝ https://data.storage.market/api/providers/f01611097
➝ https://data.storage.market/api/providers/f01208862
➝ https://data.storage.market/api/providers/f01662356
➝ https://data.storage.market/api/providers/f01423116
➝ https://data.storage.market/api/providers/f017665
➝ https://data.storage.market/api/providers/f01666984
➝ https://data.storage.market/api/providers/f0440429
➝ https://data.storage.market/api/providers/f01028552
➝ https://data.storage.market/api/providers/f0707721
➝ https://data.storage.market/api/providers/f099608
➝ https://data.storage.market/api/providers/f0142637
➝ https://data.storage.market/api/providers/f01127678
➝ https://data.storage.market/api/providers/f0501283
➝ https://data.storage.market/api/providers/f01133080
➝ https://data.storage.market/api/providers/f0127896
➝ https://data.storage.market/api/providers/f010617
➝ https://data.storage.market/api/providers/f097777
➝ https://data.storage.market/api/providers/f01367109
➝ https://data.storage.market/api/providers/f01225882
➝ https://data.storage.market/api/providers/f023971
➝ https://data.storage.market/api/providers/f0466405
➝ https://data.storage.market/api/providers/f01310564
➝ https://data.storage.market/api/providers/f02620
➝ https://data.storage.market/api/providers/f02576
➝ https://data.storage.market/api/providers/f01764587
➝ https://data.storage.market/api/providers/f019551
➝ https://data.storage.market/api/providers/f034258
➝ https://data.storage.market/api/providers/f0240185
➝ https://data.storage.market/api/providers/f01619524
➝ https://data.storage.market/api/providers/f01096124
➝ https://data.storage.market/api/providers/f01163272
➝ https://data.storage.market/api/providers/f01337533
➝ https://data.storage.market/api/providers/f01091851
➝ https://data.storage.market/api/providers/f01784458
➝ https://data.storage.market/api/providers/f08403

How do you plan to make deals to your storage providers

Others/custom tool

If you answered "Others/custom tool" in the previous question, enter the details here

Our new tool Delta, which we plan to release to the whole ecosystem.

Learn more at https://delta.store

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 1 year ago

Deleting comment

@kevzak hasn't the permissions to post this comment.

Please, contact the assignee of this issue.

kevzak commented 1 year ago

Datacap Request Trigger

Total DataCap requested

1PiB

Expected weekly DataCap usage rate

10TiB

Client address

f1myzxkfsadel3n4liaw6y53s525txm7fvsia4m6q

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1myzxkfsadel3n4liaw6y53s525txm7fvsia4m6q

DataCap allocation requested

5TiB

Id

4be64ba2-f17b-46e4-b318-7664940efc48

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceafekxjs26myw5nawamsmehmf3hnv73jtgnhcgqlwzaf4xppejqj2

Address

f1myzxkfsadel3n4liaw6y53s525txm7fvsia4m6q

Datacap Allocated

5.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

4be64ba2-f17b-46e4-b318-7664940efc48

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceafekxjs26myw5nawamsmehmf3hnv73jtgnhcgqlwzaf4xppejqj2

TakiChain commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedhg4sdkzqbjrlmv72eimgxmdklf7tucglmckae3frbelrepc23cm

Address

f1myzxkfsadel3n4liaw6y53s525txm7fvsia4m6q

Datacap Allocated

5.00TiB

Signer Address

f15impf3j2zcaex4lhyxndxswuuhv24vzstuqtxsi

Id

4be64ba2-f17b-46e4-b318-7664940efc48

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedhg4sdkzqbjrlmv72eimgxmdklf7tucglmckae3frbelrepc23cm

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

clriesco commented 1 year ago

Removed stale label and reopened issue :)

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1myzxkfsadel3n4liaw6y53s525txm7fvsia4m6q

DataCap allocation requested

20TiB

Id

b6fec6a2-4161-4719-9d9b-a2ae3dcfbf68

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1myzxkfsadel3n4liaw6y53s525txm7fvsia4m6q

Rule to calculate the allocation request amount

200% of weekly dc amount requested

DataCap allocation requested

20TiB

Total DataCap granted for client so far

45.5YiB

Datacap to be granted to reach the total amount requested by the client (1PiB)

45.5YiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
0 0 5TiB NaN 10TiB
github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

cryptowhizzard commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecj3m2klj4lsougb5v7qwzvjow4qgawecjk2q6y2b6ev7pzw6y5ty

Address

f1myzxkfsadel3n4liaw6y53s525txm7fvsia4m6q

Datacap Allocated

20.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

b6fec6a2-4161-4719-9d9b-a2ae3dcfbf68

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecj3m2klj4lsougb5v7qwzvjow4qgawecjk2q6y2b6ev7pzw6y5ty

liyunzhi-666 commented 1 year ago

checker:manualTrigger

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

-- Commented by Stale Bot.