[DataCap Application] DAG House - Web3.Storage

dchoi27 commented 11 months ago

Data Owner Name

DAG House and the users of our tools

What is your role related to the dataset

Data onramp entity that provides data onboarding services to multiple clients

Data Owner Country/Region

United States

Data Owner Industry

Web3 / Crypto

Website

https://web3.storage/

Social Media

https://twitter.com/web3storage
https://www.linkedin.com/company/dag-house

Total amount of DataCap being requested

15PiB

Expected size of single dataset (one copy)

500TiB and growing

Number of replicas to store

10

Weekly allocation of DataCap requested

250TiB

On-chain address for first allocation

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

Data Type of Application

Public, Open Commercial/Enterprise

Custom multisig

[ ] Use Custom Multisig

Identifier

No response

Share a brief history of your project and organization

DAG House was founded in 2021 as a team inside Protocol Labs to develop tools to make it easy for developers and end users to host content addressed data and store the data on Filecoin. Since then our two flagship products, NFT.Storage (Internet Archive of NFTs) and Web3.Storage (developer storage platform) are used by many prominent projects and companies in Web3 and outside of it.

Is this project associated with other projects/ecosystem stakeholders?

Yes

If answered yes, what are the other projects/ecosystem stakeholders

This project is associated with Protocol Labs. Projects have plans for independence.

Describe the data being stored onto Filecoin

User uploads that meet or adhere to web3.storage's terms of service (https://web3.storage/terms).

Where was the data currently stored in this dataset sourced from

Other

If you answered "Other" in the previous question, enter the details here

User uploads (generally for their web3 apps)

If you are a data preparer. What is your location (Country/Region)

None

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

No response

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

The goal is to be able to use the Filecoin copies as the only available copies on the network (rather than also storing the data on centralized infra), which requires things like retrieval to have high performance and global availability. As a result, some parts of the dataset have already been stored but not to the replication limit with other Datacap apps (https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1838, https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/2110).

Please share a sample of the data

This platform serves all different kinds of media, including images, files, and videos. Some examples:

https://ipfs.io/ipfs/bafybeid5jpdqzlb4tqsd6peoa7qstoxat3ovsg62wutyp4gnzqbqsggfsq
https://ipfs.io/ipfs/bafybeihity6bx24npzvvkzopjbat25ekefjwmnshe7rvldy72dxngzf644
https://ipfs.io/ipfs/bafybeicvcevx3ktiqjsfwnjguu4lnzejlhgb35brayuod5xdtn7demfdhe

Confirm that this is a public dataset that can be retrieved by anyone on the Network

[X] I confirm

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Daily

For how long do you plan to keep this dataset stored on Filecoin

Permanently

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, Africa, North America, South America, Europe, Australia (continent)

How will you be distributing your data to storage providers

HTTP or FTP server

How do you plan to choose storage providers

Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

We plan to use Spade for SP selection and deal execution. Spade is being developed by the Data Programs team at Protocol Labs (https://github.com/data-preservation-programs/spade) and supports storage client to SP matching based on requirements like geography, size, retrievability, etc.

If you already have a list of storage providers to work with, fill out their names and provider IDs below

No response

How do you plan to make deals to your storage providers

Others/custom tool

If you answered "Others/custom tool" in the previous question, enter the details here

We plan to use Spade for deal execution to onboard data to Filecoin. Spade is being developed by the Data Programs team at Protocol Labs (https://github.com/data-preservation-programs/spade).

Spade was initially servicing Slingshot deals and was referred to as the Evergreen Dealer.

Can you confirm that you will follow the Fil+ guideline

Yes

simonkim0515 commented 11 months ago

Datacap Request Trigger

Total DataCap requested

15PiB

Expected weekly DataCap usage rate

250TiB

Client address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

large-datacap-requests[bot] commented 11 months ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

DataCap allocation requested

125TiB

Id

1a683d47-7859-4670-9afc-b28fb0611030

cryptowhizzard commented 11 months ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceawtxdf2sqbes7dp3s56jqt7qyabxxz5tkpzlncsghtqz3hvlfdq2

Address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

Datacap Allocated

125.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

1a683d47-7859-4670-9afc-b28fb0611030

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceawtxdf2sqbes7dp3s56jqt7qyabxxz5tkpzlncsghtqz3hvlfdq2

liyunzhi-666 commented 11 months ago

I have been following and supporting the program before. I can continue to support it this time as well. Looking forward to excellent performance afterwards.

liyunzhi-666 commented 11 months ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacec7gqldor3rgjulokq63qrmvytxiu45sblysrmijrodkybmyu2usu

Address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

Datacap Allocated

125.00TiB

Signer Address

f1pszcrsciyixyuxxukkvtazcokexbn54amf7gvoq

Id

1a683d47-7859-4670-9afc-b28fb0611030

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec7gqldor3rgjulokq63qrmvytxiu45sblysrmijrodkybmyu2usu

large-datacap-requests[bot] commented 11 months ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

DataCap allocation requested

250TiB

Id

735ec2b8-7292-48e0-8bef-5f41eb534dff

Patapon0702 commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Overall Graphsync retrieval success rate:
Overall HTTP retrieval success rate: 2.22%
Overall Bitswap retrieval success rate:

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 97.75% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

Patapon0702 commented 11 months ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecptph7iozs3gjlsgwde4vfsqikihobcf6sbnayvatdykicmwwgj4

Address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

Datacap Allocated

250.00TiB

Signer Address

f1ho2liobpznr7llma6xcl7jtififsfvhdnudn4yy

Id

735ec2b8-7292-48e0-8bef-5f41eb534dff

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecptph7iozs3gjlsgwde4vfsqikihobcf6sbnayvatdykicmwwgj4

cryptowhizzard commented 11 months ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecbnk3mre6r7obslof4tmextmcgjfimkxhh5bpmymt5ig7pvqbooa

Address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

Datacap Allocated

250.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

735ec2b8-7292-48e0-8bef-5f41eb534dff

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecbnk3mre6r7obslof4tmextmcgjfimkxhh5bpmymt5ig7pvqbooa

large-datacap-requests[bot] commented 11 months ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

DataCap allocation requested

500TiB

Id

2bf30b49-1406-48e8-889d-2bae95955d8d

cryptowhizzard commented 11 months ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecy5dojl23opqqhgey76fs5nxa45znrarwtzpnad55iyng5m3muxs

Address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

Datacap Allocated

500.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

2bf30b49-1406-48e8-889d-2bae95955d8d

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecy5dojl23opqqhgey76fs5nxa45znrarwtzpnad55iyng5m3muxs

jamerduhgamer commented 11 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 11 months ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Overall Graphsync retrieval success rate:
Overall HTTP retrieval success rate: 0.74%
Overall Bitswap retrieval success rate:

Storage Provider Distribution

⚠️ 1 storage providers sealed more than 50% of total datacap - f010202: 59.16%

Deal Data Replication

⚠️ 97.45% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

jamerduhgamer commented 11 months ago

The # of SPs sealing the dataset has increased. f010202 has more than 50% datacap but it is only ~46 TiBs.

Willing to support this next tranche of datacap.

jamerduhgamer commented 11 months ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecmucctg7dnihxwcu7w2t73y6gmriixijhqz2zh64b6dopj3wnpko

Address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

Datacap Allocated

500.00TiB

Signer Address

f1kqdiokoeubyse4qpihf7yrpl7czx4qgupx3eyzi

Id

2bf30b49-1406-48e8-889d-2bae95955d8d

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecmucctg7dnihxwcu7w2t73y6gmriixijhqz2zh64b6dopj3wnpko

dannyob commented 11 months ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedg7jwfsjetslzyboturqjrkerzikspcsgal6efhr5djjfozteqcw

Address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

Datacap Allocated

500.00TiB

Signer Address

f1k6wwevxvp466ybil7y2scqlhtnrz5atjkkyvm4a

Id

2bf30b49-1406-48e8-889d-2bae95955d8d

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedg7jwfsjetslzyboturqjrkerzikspcsgal6efhr5djjfozteqcw

Patapon0702 commented 11 months ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceb7ncd6iws2h53bjwzw226ezm2jgv4l6afkvmo5ox3n7rj26laihu

Address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

Datacap Allocated

500.00TiB

Signer Address

f1ho2liobpznr7llma6xcl7jtififsfvhdnudn4yy

Id

2bf30b49-1406-48e8-889d-2bae95955d8d

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceb7ncd6iws2h53bjwzw226ezm2jgv4l6afkvmo5ox3n7rj26laihu

github-actions[bot] commented 10 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

dchoi27 commented 10 months ago

keep open

github-actions[bot] commented 10 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

dchoi27 commented 10 months ago

keep open

github-actions[bot] commented 10 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

dchoi27 commented 9 months ago

leave open

github-actions[bot] commented 9 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

dchoi27 commented 9 months ago

keep open

github-actions[bot] commented 9 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

dchoi27 commented 9 months ago

keep open

Sunnyiscoming commented 9 months ago

Hello, @dchoi27 per the https://github.com/filecoin-project/notary-governance/issues/922 for Open, Public Dataset applicants, please complete the following Fil+ registration form to identify yourself as the applicant and also please add the contact information of the SP entities you are working with to store copies of the data.

This information will be reviewed by Fil+ Governance team to confirm validity and then the application will be allowed to move forward for additional notary review.

aggregation-and-compliance-bot[bot] commented 8 months ago

Client f02759235 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 7 days. Please take appropiate action to fix the following DataCap usage problems.	Criteria	Treshold	Reason
Shared data percent	< 20%	23.53% of the clients data is shared with other clients. This should be less than 20%

github-actions[bot] commented 8 months ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

-- Commented by Stale Bot.

dkkapur commented 8 months ago

@Sunnyiscoming @kevzak @panges2 this one is an example of an application which is offloading SP decision making to an external service, i.e., spade in this case. Where does that fit into the setup for the registration form + AC bot?

kevzak commented 8 months ago

Hello, @dchoi27 per the filecoin-project/notary-governance#922 for Open, Public Dataset applicants, please complete the following Fil+ registration form to identify yourself as the applicant and also please add the contact information of the SP entities you are working with to store copies of the data.

This information will be reviewed by Fil+ Governance team to confirm validity and then the application will be allowed to move forward for additional notary review.

@Sunnyiscoming this applicant is using SPADE to validate and choose SPs so we do not need to manually check. Thanks

aggregation-and-compliance-bot[bot] commented 8 months ago

Client f02759235 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 14 days. Please take appropiate action to fix the following DataCap usage problems.	Criteria	Treshold	Reason
Shared data percent	< 20%	22.55% of the clients data is shared with other clients. This should be less than 20%

aggregation-and-compliance-bot[bot] commented 8 months ago

Client f02759235 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 21 days. Please take appropiate action to fix the following DataCap usage problems.	Criteria	Treshold	Reason
Shared data percent	< 20%	22.15% of the clients data is shared with other clients. This should be less than 20%

aggregation-and-compliance-bot[bot] commented 8 months ago

Client f02759235 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 28 days. Please take appropiate action to fix the following DataCap usage problems.	Criteria	Treshold	Reason
Shared data percent	< 20%	21.48% of the clients data is shared with other clients. This should be less than 20%

aggregation-and-compliance-bot[bot] commented 7 months ago

Client f02759235 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 35 days. Please take appropiate action to fix the following DataCap usage problems.	Criteria	Treshold	Reason
Shared data percent	< 20%	21% of the clients data is shared with other clients. This should be less than 20%

large-datacap-requests[bot] commented 7 months ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f02049625

Client address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

DataCap allocation requested

1000.0TiB

Id

e36061ee-9538-43dd-af79-463e149a8bb9

aggregation-and-compliance-bot[bot] commented 7 months ago

Client f02759235 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 43 days. Please take appropiate action to fix the following DataCap usage problems.	Criteria	Treshold	Reason
Shared data percent	< 20%	20.57% of the clients data is shared with other clients. This should be less than 20%

cryptowhizzard commented 7 months ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacec7psbzkcrlviwmt7y7e2rjxeifqvtjis6kxlorhf5spalrci4gt6

Address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

Datacap Allocated

1000.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

e36061ee-9538-43dd-af79-463e149a8bb9

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec7psbzkcrlviwmt7y7e2rjxeifqvtjis6kxlorhf5spalrci4gt6

liyunzhi-666 commented 7 months ago

Client f02759235 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 43 days. Please take appropiate action to fix the following DataCap usage problems.

Criteria Treshold Reason Shared data percent < 20% 20.57% of the clients data is shared with other clients. This should be less than 20%

Hi @dchoi27 Can you explain this?

dchoi27 commented 7 months ago

Nothing has changed from before - we use Spade to make deals publicly, and any SP is available to onboard our data that way. Ideally the distribution is flatter, but if not, it's more a reflection of the willingness of SPs to onboard data rather than anything we're doing.

liyunzhi-666 commented 7 months ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebz6n5jmluohloyfvic4ko4hmmli7bclxcjlblrpu72chm23f6624

Address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

Datacap Allocated

1000.00TiB

Signer Address

f1pszcrsciyixyuxxukkvtazcokexbn54amf7gvoq

Id

e36061ee-9538-43dd-af79-463e149a8bb9

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebz6n5jmluohloyfvic4ko4hmmli7bclxcjlblrpu72chm23f6624

large-datacap-requests[bot] commented 7 months ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f02049625

Client address

f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusxgurq

DataCap allocation requested

1000.0TiB

Id

0b3d4989-4f94-49a7-8b87-f19b6fa65cfc

aggregation-and-compliance-bot[bot] commented 7 months ago

Client f02759235 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 50 days. Please take appropiate action to fix the following DataCap usage problems.	Criteria	Treshold	Reason
Shared data percent	< 20%	20.06% of the clients data is shared with other clients. This should be less than 20%

github-actions[bot] commented 7 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

dchoi27 commented 7 months ago

Do not close

github-actions[bot] commented 6 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

dchoi27 commented 6 months ago

keep open

github-actions[bot] commented 6 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

filecoin-project / filecoin-plus-large-datasets