search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] NFT.Storage - NFT.Storage #2110

Open dchoi27 opened 1 year ago

dchoi27 commented 1 year ago

Data Owner Name

NFT.Storage

What is your role related to the dataset

Data Preparer

Data Owner Country/Region

United States

Data Owner Industry

Life Science / Healthcare

Website

https://nft.storage

Social Media

https://twitter.com/NFTdotStorage

Total amount of DataCap being requested

15PiB

Expected size of single dataset (one copy)

500TiB and growing

Number of replicas to store

10

Weekly allocation of DataCap requested

250TiB

On-chain address for first allocation

f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq

Data Type of Application

Public, Open Commercial/Enterprise

Custom multisig

Identifier

No response

Share a brief history of your project and organization

NFT.Storage was created in 2021 to be an "Internet Archive" of NFTs, storing off-chain NFT data as a public good using the Filecoin network.

Is this project associated with other projects/ecosystem stakeholders?

Yes

If answered yes, what are the other projects/ecosystem stakeholders

This project is associated with Protocol Labs. Projects have plans for independence.

Describe the data being stored onto Filecoin

User uploaded data that, per the NFT.Storage Terms of Service (https://nft.storage/terms), must be legitimate off-chain NFT data. In addition, NFT.Storage is likely to scrape NFT data using on-chain references in the future.

Where was the data currently stored in this dataset sourced from

Other

If you answered "Other" in the previous question, enter the details here

User uploads

If you are a data preparer. What is your location (City and Country)

Seattle, WA

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

The data preparer is DAG House, which builds client tools and runs infrastructure for NFT.Storage and web3.storage. Users upload CAR files directly, which are then aggregated into sector sized CAR files. Our client libraries do the CAR conversion for users. One example can be found here: https://github.com/web3-storage/w3up-cli

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

Sort of. We previously stored NFT.Storage data using the same Datacap wallet found here (https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1838). As copies of data expire that were stored using this other wallet, we will use this wallet to re-up the number of replicas.

Please share a sample of the data

- https://ipfs.io/ipfs/bafybeid5jpdqzlb4tqsd6peoa7qstoxat3ovsg62wutyp4gnzqbqsggfsq
- https://ipfs.io/ipfs/bafybeihity6bx24npzvvkzopjbat25ekefjwmnshe7rvldy72dxngzf644
- https://ipfs.io/ipfs/bafybeicvcevx3ktiqjsfwnjguu4lnzejlhgb35brayuod5xdtn7demfdhe

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Daily

For how long do you plan to keep this dataset stored on Filecoin

Permanently

In which geographies do you plan on making storage deals

Asia other than Greater China, Africa, North America, South America, Europe, Australia (continent)

How will you be distributing your data to storage providers

Cloud storage (i.e. S3)

How do you plan to choose storage providers

Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

We plan to use Spade for SP selection and deal execution. Spade is being developed by the Data Programs team at Protocol Labs (https://github.com/data-preservation-programs/spade) and supports storage client to SP mtching based on requirements like geography, size, retrievability, etc.

If you already have a list of storage providers to work with, fill out their names and provider IDs below

No response

How do you plan to make deals to your storage providers

Others/custom tool

If you answered "Others/custom tool" in the previous question, enter the details here

https://github.com/web3-storage/w3filecoin

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!
kevzak commented 1 year ago

@dchoi27 the bot is confused here. How much DataCap are you requesting? Need a number. If its greater than 15PiB it will go through E-Fil+ pathway. Let me know.

dchoi27 commented 1 year ago

Got it - I changed to 15PiB. (note that greater than 15PiB is listed as an option in the issue creation form)

kevzak commented 1 year ago

Yes, if you need greater than 15PiB, we can add that. Just let me know the amount. Need to do it manually so that I am triggered to work with the large data client via E-Fil processes.

Seems like for now you have 500TiB x 10 copies so ~ 5 to 6 PiB needed?

Sunnyiscoming commented 1 year ago

Any update?

dchoi27 commented 1 year ago

@Sunnyiscoming what kind of update were you looking for?

ghost commented 1 year ago

@dchoi27 can you confirm answer to question above? https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/2110#issuecomment-1650326985

dchoi27 commented 1 year ago

This one? (think the comment was edited to include this question after i adjusted the application)

Seems like for now you have 500TiB x 10 copies so ~ 5 to 6 PiB needed?

The dataset is constantly growing because of new user uploads, so we were advised to submit an application for 15PiB (so we wouldn't need to keep submitting new applications)

kevzak commented 1 year ago

Datacap Request Trigger

Total DataCap requested

15PiB

Expected weekly DataCap usage rate

250TiB

Client address

f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq

DataCap allocation requested

125TiB

Id

d467872a-eb72-48c8-babb-26c195b92111

MEIYAN666 commented 1 year ago

Official application, looks good to me, willing to support!

MEIYAN666 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebay6asyijh47npwup5bzj52xsgjtcflcmg4hxfkwxnu4myubtghq

Address

f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq

Datacap Allocated

125.00TiB

Signer Address

f1bwugfihrmn3iyunzyxst5nttql3dge4khwmurtq

Id

d467872a-eb72-48c8-babb-26c195b92111

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebay6asyijh47npwup5bzj52xsgjtcflcmg4hxfkwxnu4myubtghq

cryptowhizzard commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedx37eecvx22xgir26ws7yrxeiqyo6qeqsfu2ctjjwwwciy66f7fa

Address

f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq

Datacap Allocated

125.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

d467872a-eb72-48c8-babb-26c195b92111

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedx37eecvx22xgir26ws7yrxeiqyo6qeqsfu2ctjjwwwciy66f7fa

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

dchoi27 commented 1 year ago

This is still active

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

-- Commented by Stale Bot.

clriesco commented 1 year ago

Removed stale label, reopened issue :)

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 6

Multisig Notary address

f02049625

Client address

f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq

DataCap allocation requested

1000.0TiB

Id

4854e052-0cfa-45d4-8435-74cbee291263

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq

Rule to calculate the allocation request amount

400% of weekly dc amount requested

DataCap allocation requested

1000.0TiB

Total DataCap granted for client so far

1.1368683772161625e+40YiB

Datacap to be granted to reach the total amount requested by the client (15PiB)

1.1368683772161625e+40YiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
2678 10 125TiB 59.97 31.40TiB
cryptowhizzard commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedgiibl2py7mqgkxrs7drqnv5xep63b2ojpvoqszomt7tobdznnoq

Address

f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq

Datacap Allocated

1000.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

4854e052-0cfa-45d4-8435-74cbee291263

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedgiibl2py7mqgkxrs7drqnv5xep63b2ojpvoqszomt7tobdznnoq

spaceT9 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 96.29% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

dchoi27 commented 1 year ago

keep open

liyunzhi-666 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 96.27% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

liyunzhi-666 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacea5m5xuibh2lrlibyl3dxxbxiubdatxsat52q2hu6fwwnqc2x73wc

Address

f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq

Datacap Allocated

1000.00TiB

Signer Address

f1pszcrsciyixyuxxukkvtazcokexbn54amf7gvoq

Id

4854e052-0cfa-45d4-8435-74cbee291263

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea5m5xuibh2lrlibyl3dxxbxiubdatxsat52q2hu6fwwnqc2x73wc

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

-- Commented by Stale Bot.

dchoi27 commented 1 year ago

@simonkim0515 can we reopen

cryptowhizzard commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebehl3fo6ao777cdjf2boncpr6xrbjfeahw3hoho4ncpqmjrtwhbu

Address

f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq

Datacap Allocated

1000.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

4854e052-0cfa-45d4-8435-74cbee291263

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebehl3fo6ao777cdjf2boncpr6xrbjfeahw3hoho4ncpqmjrtwhbu

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

dchoi27 commented 1 year ago

bump

Sunnyiscoming commented 1 year ago

Hello, @dchoi27 per the https://github.com/filecoin-project/notary-governance/issues/922 for Open, Public Dataset applicants, please complete the following Fil+ registration form to identify yourself as the applicant and also please add the contact information of the SP entities you are working with to store copies of the data.

This information will be reviewed by Fil+ Governance team to confirm validity and then the application will be allowed to move forward for additional notary review.

kevzak commented 1 year ago

@Sunnyiscoming they are using Spade tooling to manage SP distribution.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

dchoi27 commented 1 year ago

keep open

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

dchoi27 commented 1 year ago

keep open

dannyob commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebjx2wjusc373gjpedpjkvi2xx5e3tdvs2z6rmawbdrpc6iips7rc

Address

f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq

Datacap Allocated

1000.00TiB

Signer Address

f1k6wwevxvp466ybil7y2scqlhtnrz5atjkkyvm4a

Id

4854e052-0cfa-45d4-8435-74cbee291263

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebjx2wjusc373gjpedpjkvi2xx5e3tdvs2z6rmawbdrpc6iips7rc

aggregation-and-compliance-bot[bot] commented 12 months ago
Client f01151139 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 7 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Cid Checker score > 25% The client has a CID checker score of 0%. This should be greater than 25%. To find out more about CID checker score please look at this issue: https://github.com/filecoin-project/notary-governance/issues/986
ghost commented 12 months ago

checker:manualTrigger

filplus-checker-app[bot] commented 12 months ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 96.35% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard.

aggregation-and-compliance-bot[bot] commented 11 months ago
Client f01151139 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 14 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Cid Checker score > 25% The client has a CID checker score of 0%. This should be greater than 25%. To find out more about CID checker score please look at this issue: https://github.com/filecoin-project/notary-governance/issues/986
aggregation-and-compliance-bot[bot] commented 11 months ago
Client f01151139 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 21 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Cid Checker score > 25% The client has a CID checker score of 0%. This should be greater than 25%. To find out more about CID checker score please look at this issue: https://github.com/filecoin-project/notary-governance/issues/986
aggregation-and-compliance-bot[bot] commented 11 months ago
Client f01151139 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 28 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Cid Checker score > 25% The client has a CID checker score of 0%. This should be greater than 25%. To find out more about CID checker score please look at this issue: https://github.com/filecoin-project/notary-governance/issues/986
aggregation-and-compliance-bot[bot] commented 11 months ago
Client f01151139 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 35 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Cid Checker score > 25% The client has a CID checker score of 0%. This should be greater than 25%. To find out more about CID checker score please look at this issue: https://github.com/filecoin-project/notary-governance/issues/986
aggregation-and-compliance-bot[bot] commented 10 months ago
Client f01151139 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 43 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Cid Checker score > 25% The client has a CID checker score of 0%. This should be greater than 25%. To find out more about CID checker score please look at this issue: https://github.com/filecoin-project/notary-governance/issues/986
aggregation-and-compliance-bot[bot] commented 10 months ago
Client f01151139 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 50 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Cid Checker score > 25% The client has a CID checker score of 0%. This should be greater than 25%. To find out more about CID checker score please look at this issue: https://github.com/filecoin-project/notary-governance/issues/986
elizabeth-griffiths commented 10 months ago

Hi - This is still active. Can you please advice the impact of failing requirements? Do you need anything from us?

On Sat, Jan 13, 2024 at 11:14 PM aggregation-and-compliance-bot[bot] < @.***> wrote:

Client f01151139 does not follow the datacap usage rules. More info here https://github.com/filecoin-project/notary-governance/issues/986. This application has been failing the requirements for 50 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason Cid Checker score > 25% The client has a CID checker score of 0%. This should be greater than 25%. To find out more about CID checker score please look at this issue: filecoin-project/notary-governance#986 https://github.com/filecoin-project/notary-governance/issues/986

— Reply to this email directly, view it on GitHub https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/2110#issuecomment-1890435326, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4YFHQPFRRTPQ7UAFETGRNTYOJ3B5AVCNFSM6AAAAAA2WNHBC6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJQGQZTKMZSGY . You are receiving this because you are subscribed to this thread.Message ID: <filecoin-project/filecoin-plus-large-datasets/issues/2110/1890435326@ github.com>

--

Elizabeth Griffiths

@.***

Product Manager

NFT.Storage | Protocol Labs