[DataCap Application] Ghost Byte Inc - Common Crawl [10/16]

[Trevor-K-Smith]

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

DP Info

• Organization Name: Ghost Byte Inc
• Website / Social Media: https://ghostbyte.io/
• Total amount of DataCap being requested (between 500 TiB and 5 PiB): 5 PiB
• Weekly allocation of DataCap requested (usually between 1-100TiB): 512 TiB
• On-chain address for first allocation: f1pn5v3fzm4rcmbaoneamigvnlwf3his6rpneqaoa

Client Info

• Organization Name: Common Crawl
• Website / Social Media: Common Crawl

Data Info

• Dataset Size: 7.51 PiB
• Replications: 10x
• Total Datacap Request: 80 PiB

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Share a brief history of your project and organization.

Ghost Byte Inc is a storage provider seeking to onboard data to meet the high demand of FIL+ for itself and its partners. Ghost Byte has a history of actively participating in NA weekly calls, helping community members on the slack channel, testing beta software with feedback, and overall ongoing support in the community of filecoin. Ghost Byte works with industry partners to assist the growth in web3 adoption.
Ref: https://www.youtube.com/watch?v=6PejYUlN0AM

What is the primary source of funding for this project?

Ghost Byte Inc

What other projects/ecosystem stakeholders is this project associated with?

Ghost Byte Inc

Use-case details

Describe the data being stored onto Filecoin

A corpus of web crawl data composed of over 50 billion web pages.

In addition, we also publish [statistics and basic metrics](https://commoncrawl.github.io/cc-crawl-statistics/) of each crawl that include:

Size of the crawl as numbers of fetched pages, unique URLs, unique documents (by content digest), number of different hosts, domains, and top-level domains
Distribution of pages/URLs on hosts, domains, top-level domains
Content language, MIME types, character sets

Where was the data in this dataset sourced from?

This data is being replicated from AWS Opendata to Filecoin. The dataset being replicated is Common Crawl. This dataset is 7.51 PiB Total Size. The data will be replicated a total of 10 times for a total datacap request of 80 PiB.
Ref: https://registry.opendata.aws/commoncrawl/

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this.

https://www.dropbox.com/scl/fo/ghig31f0gw196c2kjsv0i/h?dl=0&rlkey=n8qh6hzbxbgm3gbmb1djgdfum

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

Public - https://registry.opendata.aws/commoncrawl/

What is the expected retrieval frequency for this data?

1-3 Year

For how long do you plan to keep this dataset stored on Filecoin?

540 Days, subject to renewal when the time comes.

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

Global partners. No restrictions, will spread out as much as possible.

How will you be distributing your data to storage providers? Is there an offline data transfer process?

Data will be send over boostd to participating storage providers. Otherwise, offline deals can be done for those with special requirements.

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

Storage Providers will be found in the active community slack, partners met at events, and online data sources. 1 per actor, 2 per organization, spread as evenly across the globe a possible. Total 10 replications. SP's will be confirmed ahead of replication of cars that they intend to allow the cars to be accessible and retrievable. Car files for this replication will not be enforced to keep unsealed sectors as this is disaster recovery data, and retrievals will be low. 

How will you be distributing deals across storage providers?

Singularity will be used to serve up the deals and track the progress of each CAR file being replicated. 1 per actor, 2 per organization, spread as evenly across the globe a possible. Total 10 replications.

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

Yes, we have the resources to get started right away. We do not need help at this time. Thank you!

[large-datacap-requests[bot]]

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

[large-datacap-requests[bot]]

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

[large-datacap-requests[bot]]

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

[large-datacap-requests[bot]]

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

[simonkim0515]

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

250TiB

Client address

f13imatde7bhvicy3ps5m2reu3usdsfaf4hnirnbq

[large-datacap-requests[bot]]

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1pn5v3fzm4rcmbaoneamigvnlwf3his6rpneqaoa

DataCap allocation requested

125TiB

Id

9a5bd0b6-8f4c-45e3-a5ef-8964274efab9

[filplus-checker-app[bot]]

DataCap and CID Checker Report[^1]

There is no previous allocation for this issue.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[large-datacap-requests[bot]]

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

[Trevor-K-Smith]

@simonkim0515 @raghavrmadya

Can i kindly get this application reviewed? Thanks

[large-datacap-requests[bot]]

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

[large-datacap-requests[bot]]

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

[data-programs]

This user’s identity has been verified through filplus.storage