[DataCap Application] Outercore - Delta Instance Two

jimmylee commented 1 year ago

Data Owner Name

Outercore - Network Growth - Engineering

Data Owner Country/Region

United States

Data Owner Industry

Other

Website

https://fw.services

Social Media

https://twitter.com/OutercoreEng
https://twitter.com/Estuary_Tech

Total amount of DataCap being requested

2PiB

Weekly allocation of DataCap requested

500TiB

On-chain address for first allocation

f1p3l3wgnfukemmaupqecwcoqp7fcgjcqgqcq7rja

Custom multisig

[ ] Use Custom Multisig

Identifier

No response

Share a brief history of your project and organization

Root website: https://fw.services
Project website: https://delta.store

We are the Engineering team at Protocol Labs ➝ Outercore ➝ Engineering/Network Growth. Our goal is develop tooling for entire Filecoin Ecosystem to use to onboard data to the Filecoin Network.

This is a second application for our tool Delta. The first one is located here. https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1601

#### Our mission

We want an internet where resources are owned & shared by everyone. Everyone shares consensus over a distributed ledger. Data storage is verifiably stored for the end-user, and more fault tolerant. There is no central authority of control and no single point of failure. And the security promise only improves as the network continues to grow. Just imagine all of the certifications that can be automatically generated for end users by a new network like this. This is how you achieve storage as a human right for anyone in the world.

We're going to prove that we can get here by building the tools that can support onboarding 10 PiBs a day of data to the Filecoin Network. But we're also not going to forget about retrievability and helping the Filecoin Network actually become usable.

#### Delta

Our solution to archival and cold storage use cases.

Use ∆ Delta to upload all of your useful public data to Filecoin storage providers. Delta is a straight-forward Filecoin storage deal making tool that manages deals, and does not do anything else. It is purely for the function of helping Storage Providers fill capacity either through online or offline methods. It is written in Go and designed to be paired well with bare-metal infrastructure.

Is this project associated with other projects/ecosystem stakeholders?

Yes

If answered yes, what are the other projects/ecosystem stakeholders

Protocol Labs -> Outercore -> Network Growth

Describe the data being stored onto Filecoin

A useful datasets that have been listed by the Network Growth team has tracked in their client tracker.

We also have our own PM going after clients such as the Cancer Imaging Archive, University Data, Climate Data, AI Data, Healthcare data, and so much more.

Where was the data currently stored in this dataset sourced from

Other

If you answered "Other" in the previous question, enter the details here

Differs from client to client

* University hard drives
* Backup services
* Cloud

How do you plan to prepare the dataset

others/custom tool

If you answered "other/custom tool" in the previous question, enter the details here

Our new tool Delta, which we plan to release to the whole ecosystem.

Learn more at https://delta.store

Please share a sample of the data

I have no examples, we'll be working through Network Growth's client list of Petabyte scale users.

Confirm that this is a public dataset that can be retrieved by anyone on the Network

[ ] I confirm

If you chose not to confirm, what was the reason

Delta is not focused on retrieval, its focused on storage onboarding. We are going to help all SPs onboard all of their data.

Estuary is focused on retrieval.

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

More than 3 years

In which geographies do you plan on making storage deals

North America

How will you be distributing your data to storage providers

Others

How do you plan to choose storage providers

Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

We will be developing our tool Delta

Learn more at https://delta.store

If you already have a list of storage providers to work with, fill out their names and provider IDs below

We have developed our own list at https://data.storage.market but we will work with everyone/anyone. We actively work with storage providers and we have a storage provider on our team.

93 of them that we plan to start with, obviously we'll work with more.

➝ https://data.storage.market/api/providers/f0840770
➝ https://data.storage.market/api/providers/f01624021
➝ https://data.storage.market/api/providers/f01806491
➝ https://data.storage.market/api/providers/f08399
➝ https://data.storage.market/api/providers/f01790264
➝ https://data.storage.market/api/providers/f0875769
➝ https://data.storage.market/api/providers/f01035680
➝ https://data.storage.market/api/providers/f033356
➝ https://data.storage.market/api/providers/f01683871
➝ https://data.storage.market/api/providers/f03488
➝ https://data.storage.market/api/providers/f030379
➝ https://data.storage.market/api/providers/f01466075
➝ https://data.storage.market/api/providers/f02301
➝ https://data.storage.market/api/providers/f010479
➝ https://data.storage.market/api/providers/f010088
➝ https://data.storage.market/api/providers/f0773157
➝ https://data.storage.market/api/providers/f0717969
➝ https://data.storage.market/api/providers/f0461791
➝ https://data.storage.market/api/providers/f01746964
➝ https://data.storage.market/api/providers/f01059489
➝ https://data.storage.market/api/providers/f023467
➝ https://data.storage.market/api/providers/f01392893
➝ https://data.storage.market/api/providers/f01736668
➝ https://data.storage.market/api/providers/f022352
➝ https://data.storage.market/api/providers/f01199430
➝ https://data.storage.market/api/providers/f09848
➝ https://data.storage.market/api/providers/f01222595
➝ https://data.storage.market/api/providers/f01794610
➝ https://data.storage.market/api/providers/f01443744
➝ https://data.storage.market/api/providers/f01199442
➝ https://data.storage.market/api/providers/f02401
➝ https://data.storage.market/api/providers/f01402814
➝ https://data.storage.market/api/providers/f0104671
➝ https://data.storage.market/api/providers/f01207045
➝ https://data.storage.market/api/providers/f0406703
➝ https://data.storage.market/api/providers/f01175097
➝ https://data.storage.market/api/providers/f010446
➝ https://data.storage.market/api/providers/f0724219
➝ https://data.storage.market/api/providers/f022142
➝ https://data.storage.market/api/providers/f058369
➝ https://data.storage.market/api/providers/f01201327
➝ https://data.storage.market/api/providers/f0406322
➝ https://data.storage.market/api/providers/f01278
➝ https://data.storage.market/api/providers/f0187709
➝ https://data.storage.market/api/providers/f01045784
➝ https://data.storage.market/api/providers/f0706693
➝ https://data.storage.market/api/providers/f024184
➝ https://data.storage.market/api/providers/f01385207
➝ https://data.storage.market/api/providers/f01652333
➝ https://data.storage.market/api/providers/f01319368
➝ https://data.storage.market/api/providers/f0214334
➝ https://data.storage.market/api/providers/f082635
➝ https://data.storage.market/api/providers/f0836160
➝ https://data.storage.market/api/providers/f0135078
➝ https://data.storage.market/api/providers/f039940
➝ https://data.storage.market/api/providers/f0408717
➝ https://data.storage.market/api/providers/f01345523
➝ https://data.storage.market/api/providers/f01108096
➝ https://data.storage.market/api/providers/f01611097
➝ https://data.storage.market/api/providers/f01208862
➝ https://data.storage.market/api/providers/f01662356
➝ https://data.storage.market/api/providers/f01423116
➝ https://data.storage.market/api/providers/f017665
➝ https://data.storage.market/api/providers/f01666984
➝ https://data.storage.market/api/providers/f0440429
➝ https://data.storage.market/api/providers/f01028552
➝ https://data.storage.market/api/providers/f0707721
➝ https://data.storage.market/api/providers/f099608
➝ https://data.storage.market/api/providers/f0142637
➝ https://data.storage.market/api/providers/f01127678
➝ https://data.storage.market/api/providers/f0501283
➝ https://data.storage.market/api/providers/f01133080
➝ https://data.storage.market/api/providers/f0127896
➝ https://data.storage.market/api/providers/f010617
➝ https://data.storage.market/api/providers/f097777
➝ https://data.storage.market/api/providers/f01367109
➝ https://data.storage.market/api/providers/f01225882
➝ https://data.storage.market/api/providers/f023971
➝ https://data.storage.market/api/providers/f0466405
➝ https://data.storage.market/api/providers/f01310564
➝ https://data.storage.market/api/providers/f02620
➝ https://data.storage.market/api/providers/f02576
➝ https://data.storage.market/api/providers/f01764587
➝ https://data.storage.market/api/providers/f019551
➝ https://data.storage.market/api/providers/f034258
➝ https://data.storage.market/api/providers/f0240185
➝ https://data.storage.market/api/providers/f01619524
➝ https://data.storage.market/api/providers/f01096124
➝ https://data.storage.market/api/providers/f01163272
➝ https://data.storage.market/api/providers/f01337533
➝ https://data.storage.market/api/providers/f01091851
➝ https://data.storage.market/api/providers/f01784458
➝ https://data.storage.market/api/providers/f08403

How do you plan to make deals to your storage providers

Others/custom tool

If you answered "Others/custom tool" in the previous question, enter the details here

Our new tool Delta, which we plan to release to the whole ecosystem.

Learn more at https://delta.store

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

simonkim0515 commented 1 year ago

Datacap Request Trigger

Total DataCap requested

2PiB

Expected weekly DataCap usage rate

500TiB

Client address

f1p3l3wgnfukemmaupqecwcoqp7fcgjcqgqcq7rja

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f01858410

Client address

f1p3l3wgnfukemmaupqecwcoqp7fcgjcqgqcq7rja

DataCap allocation requested

102.39TiB

Id

58c62c9d-f390-430f-b6dd-e11df78bb824

liyunzhi-666 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaced5dkws4w4cgu2nhdwdnoerzx6rekd2k2drzulxv6okamsozc22pc

Address

f1p3l3wgnfukemmaupqecwcoqp7fcgjcqgqcq7rja

Datacap Allocated

102.39TiB

Signer Address

f1pszcrsciyixyuxxukkvtazcokexbn54amf7gvoq

Id

58c62c9d-f390-430f-b6dd-e11df78bb824

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaced5dkws4w4cgu2nhdwdnoerzx6rekd2k2drzulxv6okamsozc22pc

liyunzhi-666 commented 1 year ago

I saw @jimmylee's short introduction in the slck channel as well as I checked out what was in the application, and it all worked out well for me. I'm willing to support the first round.

kernelogic commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacea4tsupn7ud24a7hzv4bdoi2ly2xtpohxlobvugq3yb5ay3f6gxto

Address

f1p3l3wgnfukemmaupqecwcoqp7fcgjcqgqcq7rja

Datacap Allocated

102.39TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

58c62c9d-f390-430f-b6dd-e11df78bb824

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea4tsupn7ud24a7hzv4bdoi2ly2xtpohxlobvugq3yb5ay3f6gxto

jimmylee commented 1 year ago

Thank you everyone <3

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1p3l3wgnfukemmaupqecwcoqp7fcgjcqgqcq7rja

DataCap allocation requested

204.79TiB

Id

a4074253-4b92-41a9-bed2-7184197d1502

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1p3l3wgnfukemmaupqecwcoqp7fcgjcqgqcq7rja

Rule to calculate the allocation request amount

10% of total dc amount requested

DataCap allocation requested

204.79TiB

Total DataCap granted for client so far

102.39TiB

Datacap to be granted to reach the total amount requested by the client (2PiB)

1.90PiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
4250	30	102.39TiB	79.74	25.66TiB

cryptowhizzard commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Overall Graphsync retrieval success rate: 39.16%
Overall HTTP retrieval success rate: 0.00%
Overall Bitswap retrieval success rate: 3.19%

Storage Provider Distribution

⚠️ 1 storage providers sealed more than 70% of total datacap - f01963614: 79.64%

⚠️ 2 storage providers sealed too much duplicate data - f01310564: 81.82%, f01059489: 42.42%

Deal Data Replication

⚠️ 99.88% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedntq4zxhbqidkv4l7ptrmb42ar5jsmqafw32d56s4eyvhj5pfqtk

Address

f1p3l3wgnfukemmaupqecwcoqp7fcgjcqgqcq7rja

Datacap Allocated

204.79TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

a4074253-4b92-41a9-bed2-7184197d1502

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedntq4zxhbqidkv4l7ptrmb42ar5jsmqafw32d56s4eyvhj5pfqtk

kernelogic commented 1 year ago

https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1854#issuecomment-1612381964

cryptowhizzard commented 1 year ago

Retrieval rate improving, known Engineering team at Protocol Labs. Received in depth Client tracking data with all institutions involved. Willing to support both applications.

@jimmylee when will this be resolved? 1 storage providers sealed more than 70% of total datacap - f01963614: 79.64%

cryptowhizzard commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecez6ynhddm6mctqagrvfmnzhjaqdsgx5vxsjh75jbjhv5a6os7iw

Address

f1p3l3wgnfukemmaupqecwcoqp7fcgjcqgqcq7rja

Datacap Allocated

204.79TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

a4074253-4b92-41a9-bed2-7184197d1502

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecez6ynhddm6mctqagrvfmnzhjaqdsgx5vxsjh75jbjhv5a6os7iw

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

filecoin-project / filecoin-plus-large-datasets