search

fidlabs / Open-Data-Pathway

6 stars 8 forks source link

[DataCap Application] w3s - web3.storage #28

Open heyjay44 opened 5 months ago

heyjay44 commented 5 months ago

Data Owner Name

w3s and the users of our tools

Data Owner Country/Region

United States

Data Owner Industry

Web3 / Crypto

Website

https://web3.storage/

Social Media Handle

https://twitter.com/web3storage

Social Media Type

Twitter

What is your role related to the dataset

Data onramp entity that provides data onboarding services to multiple clients

Total amount of DataCap being requested

4 PiB

Expected size of single dataset (one copy)

600 TiB

Number of replicas to store

10

Weekly allocation of DataCap requested

250 TiB

On-chain address for first allocation

f1rriorjgkxfktrdrjusgplm3yx4wr7cpjumol5aa

Data Type of Application

Public, Open Commercial/Enterprise

Custom multisig

Identifier

No response

Share a brief history of your project and organization

DAG House - now w3s - was founded in 2021 as a team inside Protocol Labs to develop tools to make it easy for developers and end users to host content addressed data and store the data on Filecoin. Since then our two flagship products, NFT.Storage (Internet Archive of NFTs) and Web3.Storage (developer storage platform) are used by many prominent projects and companies in Web3 and outside of it. NFT.Storage has recently spun out into its own corporate entity.

Is this project associated with other projects/ecosystem stakeholders?

Yes

If answered yes, what are the other projects/ecosystem stakeholders

This project is associated with Protocol Labs. Projects have plans for independence.

Describe the data being stored onto Filecoin

User uploads that meet or adhere to web3.storage's Terms of Service (https://web3.storage/docs/terms/).

Where was the data currently stored in this dataset sourced from

Other

If you answered "Other" in the previous question, enter the details here

User uploads (generally for their web3 apps)

If you are a data preparer. What is your location (Country/Region)

None

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

No response

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

The goal is to be able to use the Filecoin copies as the only available copies on the network (rather than also storing the data on centralized infra), which requires things like retrieval to have high performance and global availability. As a result, some parts of the dataset have already been stored but not to the replication limit with other Datacap apps (https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1838, https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/2110, https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/2192).

Please share a sample of the data

This platform serves all different kinds of media, including images, files, and videos. Some examples:
https://ipfs.io/ipfs/bafybeid5jpdqzlb4tqsd6peoa7qstoxat3ovsg62wutyp4gnzqbqsggfsq
https://ipfs.io/ipfs/bafybeihity6bx24npzvvkzopjbat25ekefjwmnshe7rvldy72dxngzf644
https://ipfs.io/ipfs/bafybeicvcevx3ktiqjsfwnjguu4lnzejlhgb35brayuod5xdtn7demfdhe

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Daily

For how long do you plan to keep this dataset stored on Filecoin

Permanently

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, Africa, North America, South America, Europe, Australia (continent)

How will you be distributing your data to storage providers

HTTP or FTP server

How did you find your storage providers

Others

If you answered "Others" in the previous question, what is the tool or platform you used

We plan to use Spade for SP selection and deal execution. Spade is managed by the Akave team, which is currently affiliated with Protocol Labs and supports storage client to SP matching based on requirements like geography, size, retrievability, etc.

Please list the provider IDs and location of the storage providers you will be working with.

web3.storage's Filecoin deals are brokered through Spade, which handles SP selection.

How do you plan to make deals to your storage providers

No response

If you answered "Others/custom tool" in the previous question, enter the details here

We plan to use Spade for deal execution to onboard data to Filecoin. Spade is being managed by the Akave team. 

Spade was initially servicing Slingshot deals and was referred to as the Evergreen Dealer.

Can you confirm that you will follow the Fil+ guideline

Yes

datacap-bot[bot] commented 5 months ago

Application is waiting for allocator review

datacap-bot[bot] commented 5 months ago

Datacap Request Trigger

Total DataCap requested

2 PiB

Expected weekly DataCap usage rate

250 TiB

DataCap Amount - First Tranche

100TiB

Client address

f1rriorjgkxfktrdrjusgplm3yx4wr7cpjumol5aa

datacap-bot[bot] commented 5 months ago

DataCap Allocation requested

Multisig Notary address

Client address

f1rriorjgkxfktrdrjusgplm3yx4wr7cpjumol5aa

DataCap allocation requested

100TiB

Id

7ca6bce9-e522-44ea-a95f-9c2d274d679e

datacap-bot[bot] commented 5 months ago

Application is ready to sign

kevzak commented 5 months ago

Client is a trusted user, eligible for 5% of total request (100TiB) for the first allocation. Just need KYC completed @heyjay44 https://filplus.storage/kyc

datacap-bot[bot] commented 5 months ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebfwtzxqk44nybueds57dazkxama6oe7ttzrkevkiqfu3trssgqyi

Address

f1rriorjgkxfktrdrjusgplm3yx4wr7cpjumol5aa

Datacap Allocated

100TiB

Signer Address

f1v24knjbqv5p6qrmfjj5xmlaoddzqnon2oxkzkyq

Id

7ca6bce9-e522-44ea-a95f-9c2d274d679e

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebfwtzxqk44nybueds57dazkxama6oe7ttzrkevkiqfu3trssgqyi

datacap-bot[bot] commented 5 months ago

Application is Granted

kevzak commented 5 months ago

Confirm KYC complete

kevzak commented 4 months ago

checker:manualTrigger

datacap-bot[bot] commented 4 months ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

⚠️ 50.00% of Storage Providers have retrieval success rate equal to zero.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report.

kevzak commented 4 months ago

Client deal making looks good, client is eligible for 15% second allocation (300TiB)

datacap-bot[bot] commented 4 months ago

Client used 75% of the allocated DataCap. Consider allocating next tranche.

datacap-bot[bot] commented 4 months ago

Client used 75% of the allocated DataCap. Consider allocating next tranche.

datacap-bot[bot] commented 4 months ago

Application is in Refill

datacap-bot[bot] commented 4 months ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceab4fntzdmq2zno73hxw7bomkdg7xhu6tcgtfpz76t2dotp5vixpm

Address

f1rriorjgkxfktrdrjusgplm3yx4wr7cpjumol5aa

Datacap Allocated

300TiB

Signer Address

f1v24knjbqv5p6qrmfjj5xmlaoddzqnon2oxkzkyq

Id

3ca7863f-0875-4b3f-afbe-1fec95b48969

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceab4fntzdmq2zno73hxw7bomkdg7xhu6tcgtfpz76t2dotp5vixpm

datacap-bot[bot] commented 4 months ago

Application is Granted

datacap-bot[bot] commented 4 months ago

Client used 75% of the allocated DataCap. Consider allocating next tranche.

datacap-bot[bot] commented 3 months ago

Client used 75% of the allocated DataCap. Consider allocating next tranche.

datacap-bot[bot] commented 3 months ago

Application is in Refill

kevzak commented 3 months ago

Deal making looks good - allocating third allocation (30%) 600TiB

datacap-bot[bot] commented 3 months ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceaa6qeguca7aqgjk7vtggtmppamtpznxl5vkzotkwpci4wdu7bk3m

Address

f1rriorjgkxfktrdrjusgplm3yx4wr7cpjumol5aa

Datacap Allocated

600TiB

Signer Address

f1v24knjbqv5p6qrmfjj5xmlaoddzqnon2oxkzkyq

Id

a883f2e9-5e97-4740-b3a9-d56502682807

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceaa6qeguca7aqgjk7vtggtmppamtpznxl5vkzotkwpci4wdu7bk3m

datacap-bot[bot] commented 3 months ago

Application is Granted

datacap-bot[bot] commented 3 months ago

Client used 75% of the allocated DataCap. Consider allocating next tranche.

martapiekarska commented 2 months ago

Bug in the bot, manually changing it back to "granted"

datacap-bot[bot] commented 2 months ago

Client used 75% of the allocated DataCap. Consider allocating next tranche.

alanshaw commented 1 month ago

@martapiekarska @kevzak is another refill possible here?

Also, https://datacapstats.io/clients?filter=f1rriorjgkxfktrdrjusgplm3yx4wr7cpjumol5aa - are we supposed to be able to monitor usage with this tool?

martplo commented 1 month ago

Sorry for the delay. I am starting to look into this issue.

Yes, the tool should be displaying how much DC you've left. There might be some issue with that. Please, create an issue here https://github.com/fidlabs/filplus-dashboard-webapp/issues, describing your problem.

martplo commented 1 month ago

checker:manualTrigger

datacap-bot[bot] commented 1 month ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

⚠️ 1 storage providers sealed more than 40% of total datacap - f01896422: 51.02%

⚠️ 14.29% of Storage Providers have retrieval success rate equal to zero.

⚠️ 42.86% of Storage Providers have retrieval success rate less than 75%.

Deal Data Replication

⚠️ 90.24% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report.

datacap-bot[bot] commented 1 month ago

Application is in Refill

datacap-bot[bot] commented 1 month ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceae3bb3b4btabkyuqjpmmwaf376dgv5vvyiclmnbrwww32ief3doe

Address

f1rriorjgkxfktrdrjusgplm3yx4wr7cpjumol5aa

Datacap Allocated

1000.0TiB

Signer Address

f1msap4wvgzzv4xlzeq6kycmgx55ferfloxnt2rcy

Id

0eac043a-2baf-46c8-a67e-7f290b1de732

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceae3bb3b4btabkyuqjpmmwaf376dgv5vvyiclmnbrwww32ief3doe

datacap-bot[bot] commented 1 month ago

Application is Granted

filecoin-watchdog commented 1 month ago

checker:manualTrigger

datacap-bot[bot] commented 1 month ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

⚠️ 1 storage providers sealed more than 40% of total datacap - f01896422: 49.71%

⚠️ 12.50% of Storage Providers have retrieval success rate equal to zero.

⚠️ 62.50% of Storage Providers have retrieval success rate less than 75%.

Deal Data Replication

⚠️ 90.20% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report.

datacap-bot[bot] commented 1 week ago

Issue has been modified. Changes below:

(OLD vs NEW)

Total Requested Amount: 4 PiB vs 2 PiB State: ChangesRequested vs Granted

datacap-bot[bot] commented 1 week ago

Issue information change request has been approved.

martplo commented 1 day ago

checker:manualTrigger

datacap-bot[bot] commented 1 day ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

⚠️ 2 storage providers sealed more than 25% of total datacap - f01896422: 38.77%, f0717969: 42.29%

⚠️ 22.22% of Storage Providers have retrieval success rate equal to zero.

⚠️ 77.78% of Storage Providers have retrieval success rate less than 75%.

Deal Data Replication

⚠️ 88.51% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report.

martplo commented 1 day ago

@heyjay44 Thank you for the updates on your application—I really appreciate the work you’re putting into this! I had a few points I wanted to discuss to ensure everything is running smoothly:

Thanks so much for your attention to these points! Let me know if you need any help or additional context—happy to support where I can.

heyjay44 commented 1 day ago

@martplo

To address all your questions: we use Spade - now Akave - as the broker for Filecoin deals so we aren't in direct control over which SPs are taking deals. We don't interface directly with them, so I'm not sure how to address your concerns.

How do you want to handle this?