filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] Outercore - Estuary (https://estuary.tech) #1603

Closed jimmylee closed 1 year ago

jimmylee commented 1 year ago

Data Owner Name

Outercore - Network Growth - Engineering

Data Owner Country/Region

United States

Data Owner Industry

Other

Website

https://estuary.tech

Social Media

https://twitter.com/OutercoreEng
https://twitter.com/Estuary_Tech

Total amount of DataCap being requested

4PiB

Weekly allocation of DataCap requested

500TiB

On-chain address for first allocation

f3vnq2cmwig3qjisnx5hobxvsd4drn4f54xfxnv4tciw6vnjdsf5xipgafreprh5riwmgtcirpcdmi3urbg36a

Custom multisig

Identifier

No response

Share a brief history of your project and organization

Root website: https://fw.services
Project website: https://delta.store

We are the Engineering team at Protocol Labs ➝ Outercore ➝ Engineering/Network Growth. Our goal is develop tooling for entire Filecoin Ecosystem to use to onboard data to the Filecoin Network.

#### Our mission

We want an internet where resources are owned & shared by everyone. Everyone shares consensus over a distributed ledger. Data storage is verifiably stored for the end-user, and more fault tolerant. There is no central authority of control and no single point of failure. And the security promise only improves as the network continues to grow. Just imagine all of the certifications that can be automatically generated for end users by a new network like this. This is how you achieve storage as a human right for anyone in the world.

We're going to prove that we can get here by building the tools that can support onboarding 10 PiBs a day of data to the Filecoin Network. But we're also not going to forget about retrievability and helping the Filecoin Network actually become usable.

### Estuary.tech

A reliable way to upload public data onto Filecoin and pin it to IPFS.

* We're building an open source platform that allows any developer to upload public data to Filecoin and retrieve it.

* Estuary.tech is free while the service is in development. We will let the public know a year in advance before we charge money for anything you depend on. For now, there is no limit to uploads, but for each file there is a 32 GiB max size.

* Check out Storage Market to see how our price compares to 300+ other Web2 or Web3 services.
The technology is developed by the Outercore: Engineering team at Protocol Labs.
Estuary.tech provides storage through immutable content address pinning. This is just like any IPFS Node, or popular services like Pinata and Web3 Storage.

* All deals have receipts before and after getting on chain. Every piece of data gets uploaded to Filecoin six times. See the example of a storage deal receipt. Each Receipt has instructions of how to retrieve.

* We need to achieve 99.99% uptime; we are working on it.

* Our goal is one hundred terabytes of data ingestion a week.

* You can contribute to the development of our technology on GitHub. Did you know every line of our service is open source? We built it all in the public. You could take everything to build your own at any time!

* Our plans for 2023.

* Stay tuned for our redesign.

* We are available 24/7 in the Filecoin Slack. Join the #ecosystem-dev channel.

* We support white-labeling. Ask us about this if you're interested.

Is this project associated with other projects/ecosystem stakeholders?

Yes

If answered yes, what are the other projects/ecosystem stakeholders

Protocol Labs -> Outercore -> Network Growth

Describe the data being stored onto Filecoin

We have a ton of meaningful applications using Estuary, check out https://estuary.tech/ecosystem to see them all.

Where was the data currently stored in this dataset sourced from

My Own Storage Infra

If you answered "Other" in the previous question, enter the details here

We run our own IPFS pinning nodes and push our own data to the Filecoin Network. All data is "useful" and "retrievable" because the focus of Estuary is to be useful to services like YouTube, Tiktok, Twitch, etc.

How do you plan to prepare the dataset

others/custom tool

If you answered "other/custom tool" in the previous question, enter the details here

Estuary does its own data preparation, its a service where all data is also retrievable, see https://estuary.tech

Please share a sample of the data

Anything you can view from any IPFS gateway.

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

I confirmed, everything that is uploaded to Estuary is retrievable, thats why https://estuary.tech exists.

What is the expected retrieval frequency for this data

Daily

For how long do you plan to keep this dataset stored on Filecoin

More than 3 years

In which geographies do you plan on making storage deals

North America

How will you be distributing your data to storage providers

Others

How do you plan to choose storage providers

Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

We have our own reputation system in Estuary where we transfer data directly from our IPFS nodes to their Filecoin Storage Providers.

If you already have a list of storage providers to work with, fill out their names and provider IDs below

We have developed our own list at https://data.storage.market but we will work with everyone/anyone. We actively work with storage providers and we have a storage provider on our team.

93 of them that we plan to start with, obviously we'll work with more.

➝ https://data.storage.market/api/providers/f0840770
➝ https://data.storage.market/api/providers/f01624021
➝ https://data.storage.market/api/providers/f01806491
➝ https://data.storage.market/api/providers/f08399
➝ https://data.storage.market/api/providers/f01790264
➝ https://data.storage.market/api/providers/f0875769
➝ https://data.storage.market/api/providers/f01035680
➝ https://data.storage.market/api/providers/f033356
➝ https://data.storage.market/api/providers/f01683871
➝ https://data.storage.market/api/providers/f03488
➝ https://data.storage.market/api/providers/f030379
➝ https://data.storage.market/api/providers/f01466075
➝ https://data.storage.market/api/providers/f02301
➝ https://data.storage.market/api/providers/f010479
➝ https://data.storage.market/api/providers/f010088
➝ https://data.storage.market/api/providers/f0773157
➝ https://data.storage.market/api/providers/f0717969
➝ https://data.storage.market/api/providers/f0461791
➝ https://data.storage.market/api/providers/f01746964
➝ https://data.storage.market/api/providers/f01059489
➝ https://data.storage.market/api/providers/f023467
➝ https://data.storage.market/api/providers/f01392893
➝ https://data.storage.market/api/providers/f01736668
➝ https://data.storage.market/api/providers/f022352
➝ https://data.storage.market/api/providers/f01199430
➝ https://data.storage.market/api/providers/f09848
➝ https://data.storage.market/api/providers/f01222595
➝ https://data.storage.market/api/providers/f01794610
➝ https://data.storage.market/api/providers/f01443744
➝ https://data.storage.market/api/providers/f01199442
➝ https://data.storage.market/api/providers/f02401
➝ https://data.storage.market/api/providers/f01402814
➝ https://data.storage.market/api/providers/f0104671
➝ https://data.storage.market/api/providers/f01207045
➝ https://data.storage.market/api/providers/f0406703
➝ https://data.storage.market/api/providers/f01175097
➝ https://data.storage.market/api/providers/f010446
➝ https://data.storage.market/api/providers/f0724219
➝ https://data.storage.market/api/providers/f022142
➝ https://data.storage.market/api/providers/f058369
➝ https://data.storage.market/api/providers/f01201327
➝ https://data.storage.market/api/providers/f0406322
➝ https://data.storage.market/api/providers/f01278
➝ https://data.storage.market/api/providers/f0187709
➝ https://data.storage.market/api/providers/f01045784
➝ https://data.storage.market/api/providers/f0706693
➝ https://data.storage.market/api/providers/f024184
➝ https://data.storage.market/api/providers/f01385207
➝ https://data.storage.market/api/providers/f01652333
➝ https://data.storage.market/api/providers/f01319368
➝ https://data.storage.market/api/providers/f0214334
➝ https://data.storage.market/api/providers/f082635
➝ https://data.storage.market/api/providers/f0836160
➝ https://data.storage.market/api/providers/f0135078
➝ https://data.storage.market/api/providers/f039940
➝ https://data.storage.market/api/providers/f0408717
➝ https://data.storage.market/api/providers/f01345523
➝ https://data.storage.market/api/providers/f01108096
➝ https://data.storage.market/api/providers/f01611097
➝ https://data.storage.market/api/providers/f01208862
➝ https://data.storage.market/api/providers/f01662356
➝ https://data.storage.market/api/providers/f01423116
➝ https://data.storage.market/api/providers/f017665
➝ https://data.storage.market/api/providers/f01666984
➝ https://data.storage.market/api/providers/f0440429
➝ https://data.storage.market/api/providers/f01028552
➝ https://data.storage.market/api/providers/f0707721
➝ https://data.storage.market/api/providers/f099608
➝ https://data.storage.market/api/providers/f0142637
➝ https://data.storage.market/api/providers/f01127678
➝ https://data.storage.market/api/providers/f0501283
➝ https://data.storage.market/api/providers/f01133080
➝ https://data.storage.market/api/providers/f0127896
➝ https://data.storage.market/api/providers/f010617
➝ https://data.storage.market/api/providers/f097777
➝ https://data.storage.market/api/providers/f01367109
➝ https://data.storage.market/api/providers/f01225882
➝ https://data.storage.market/api/providers/f023971
➝ https://data.storage.market/api/providers/f0466405
➝ https://data.storage.market/api/providers/f01310564
➝ https://data.storage.market/api/providers/f02620
➝ https://data.storage.market/api/providers/f02576
➝ https://data.storage.market/api/providers/f01764587
➝ https://data.storage.market/api/providers/f019551
➝ https://data.storage.market/api/providers/f034258
➝ https://data.storage.market/api/providers/f0240185
➝ https://data.storage.market/api/providers/f01619524
➝ https://data.storage.market/api/providers/f01096124
➝ https://data.storage.market/api/providers/f01163272
➝ https://data.storage.market/api/providers/f01337533
➝ https://data.storage.market/api/providers/f01091851
➝ https://data.storage.market/api/providers/f01784458
➝ https://data.storage.market/api/providers/f08403

How do you plan to make deals to your storage providers

Others/custom tool

If you answered "Others/custom tool" in the previous question, enter the details here

Estuary has its own online deal maker that every user depends on.

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!
large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

herrehesse commented 1 year ago

@jimmylee I'm excited to witness more growth from the Estuary team! Keep the great work going. You have my full backing.

simonkim0515 commented 1 year ago

Datacap Request Trigger

Total DataCap requested

4PiB

Expected weekly DataCap usage rate

500TiB

Client address

f3vnq2cmwig3qjisnx5hobxvsd4drn4f54xfxnv4tciw6vnjdsf5xipgafreprh5riwmgtcirpcdmi3urbg36a

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f01858410

Client address

f3vnq2cmwig3qjisnx5hobxvsd4drn4f54xfxnv4tciw6vnjdsf5xipgafreprh5riwmgtcirpcdmi3urbg36a

DataCap allocation requested

204.79TiB

Id

5e8c547d-03c5-452a-8af3-1953a592d22c

liyunzhi-666 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceaoxlh6shgicop7vkn7xfmfzciw6t2qo7kt2cckvuyug7z4owtsga

Address

f3vnq2cmwig3qjisnx5hobxvsd4drn4f54xfxnv4tciw6vnjdsf5xipgafreprh5riwmgtcirpcdmi3urbg36a

Datacap Allocated

204.79TiB

Signer Address

f1pszcrsciyixyuxxukkvtazcokexbn54amf7gvoq

Id

5e8c547d-03c5-452a-8af3-1953a592d22c

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceaoxlh6shgicop7vkn7xfmfzciw6t2qo7kt2cckvuyug7z4owtsga

liyunzhi-666 commented 1 year ago

I've been following Estuary's project progress and hope Estuary will get better and better. I am willing to support this round.

kernelogic commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebyhd5hzk3my235ayfl2ifw6hthpgvzgql2cjiiy3tkcpobo35hbc

Address

f3vnq2cmwig3qjisnx5hobxvsd4drn4f54xfxnv4tciw6vnjdsf5xipgafreprh5riwmgtcirpcdmi3urbg36a

Datacap Allocated

204.79TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

5e8c547d-03c5-452a-8af3-1953a592d22c

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebyhd5hzk3my235ayfl2ifw6hthpgvzgql2cjiiy3tkcpobo35hbc

claydrone commented 1 year ago

nebula block has been working as an SP and own f02401. Estuary is an respected core dapp project

flyworker commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceduyf2t27kswjh74sfzjwvjpzn6ziolh7svgj6nqcnslzrosvy4hw

Address

f3vnq2cmwig3qjisnx5hobxvsd4drn4f54xfxnv4tciw6vnjdsf5xipgafreprh5riwmgtcirpcdmi3urbg36a

Datacap Allocated

204.79TiB

Signer Address

f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy

Id

5e8c547d-03c5-452a-8af3-1953a592d22c

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceduyf2t27kswjh74sfzjwvjpzn6ziolh7svgj6nqcnslzrosvy4hw

dannyob commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecgi4og3iwg4tijznzy7p2j42ggw2qjcswlmp4jzxcuygzsu76spo

Address

f3vnq2cmwig3qjisnx5hobxvsd4drn4f54xfxnv4tciw6vnjdsf5xipgafreprh5riwmgtcirpcdmi3urbg36a

Datacap Allocated

204.79TiB

Signer Address

f1k6wwevxvp466ybil7y2scqlhtnrz5atjkkyvm4a

Id

5e8c547d-03c5-452a-8af3-1953a592d22c

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecgi4og3iwg4tijznzy7p2j42ggw2qjcswlmp4jzxcuygzsu76spo

jimmylee commented 1 year ago

Thank you for supporting us <3 we will work hard this year

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

aggregation-and-compliance-bot[bot] commented 10 months ago
Client f0397376 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 7 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Shared data percent < 20% 35.22% of the clients data is shared with other clients. This should be less than 20%