galen-mcandrew commented 2 years ago

Large Dataset Notary Application

To apply for a DataCap allocation for your dataset, please fill out the following information.

Core Information

Organization Name: The Internet Archive
Website / Social Media: https://archive.org/
Total amount of DataCap being requested (between 500 TiB and 5 PiB): 5PiB
Weekly allocation of DataCap requested (usually between 1-100TiB): 100TiB
On-chain address for first allocation: f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Please respond to the questions below in pargraph form, replacing the text saying "Please answer here". Include as much detail as you can in your answer!

Project details

Share a brief history of your project and organization.

The Internet Archive, a 501(c)(3) non-profit, is building a digital library of Internet sites and other cultural artifacts in digital form. Like a physical library, we provide free access to researchers, historians, scholars, the print disabled, and the general public. Our mission is to provide Universal Access to All Knowledge. See more at https://archive.org/about/

This project aims to explore the role of decentralized storage in this long-term mission.

What is the primary source of funding for this project?

We are funded through donations, grants, and by providing web archiving and book digitization services for our partners.

What other projects/ecosystem stakeholders is this project associated with?

The dataset was compiled in collaboration with The Library of Congress, California Digital Library, University of North Texas Libraries, Internet Archive, George Washington University Libraries, Stanford University Libraries, and the U.S. Government Publishing Office.

Use-case details

Describe the data being stored onto Filecoin

The End-of-Term Web Archive captures and saves U.S. Government websites at the end of presidential administrations. This dataset represents a comprehensive crawl of the .gov domain September 2016 and January 20, 2017, at the end of the Obama Administration and just before the beginning of the Trump Administration.

Where was the data in this dataset sourced from?

Federal Government websites (.gov) in the Legislative, Executive, or Judicial branches of government, and related social media accounts. Also in scope are Federal Government Websites on other domains, such as .mil, .edu, and .com

Can you share a sample of what is in the dataset? A link to a file, an image, a table, etc., are good examples of this.

The dataset contains WARC files containing crawl data (and associated metadata) of the aforementioned sites. Their contents, when opened with a compatible viewer, are similar to https://web.archive.org/web/20170126033350/http:/globalchange.epa.gov/

The raw files look like this: https://archive.org/download/LOC-QUARTERLY-006-20161225070227072-13019-13025-wbgrp-crawl202

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

Yes, data is archived in the public interest. Archive is currently available at http://eotarchive.cdlib.org/search?f1-administration=2016

What is the expected retrieval frequency for this data?

This effort is intended primarily as an archival and exploratory usecase. Data may be accessed by researchers, periodic integrity checks, and interactive use prototypes (similar to Estuary)

For how long do you plan to keep this dataset stored on Filecoin? Will this be a permanent archival or a one-time storage deal?

The dataset is intended for long-term archival storage, depending on the outcomes of this trial.

DataCap allocation plan

In which geographies do you plan on making storage deals?

We're looking for a wide geographic distribution to model global resiliency. Miners in NA and EU geos will initially be considered.

What is your expected data onboarding rate? How many deals can you make in a day, in a week? How much DataCap do you plan on using per day, per week?

We have extensive interconnects to high bandwidth networks and robust processing capacity. Once we get through the testing phase, we expect us to be able to onboard between 50-100TiB/week.

How will you be distributing your data to miners? Is there an offline data transfer process?

Offline data transfer over the internet, using standard HTTP or purose-made protocol like Tachyon.

How do you plan on choosing the miners with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

Miners that are in the right geographies and have high reputation scores on public indices like filrep.io. The initial set of storage providers for testing will likely be from the MinerX Fellowship.

How will you be distributing data and DataCap across miners storing data?

We will likely be structuring our files into 32GiB chunks that will be evenly distributed in deals with the selected set of storage providers.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

galen-mcandrew commented 2 years ago

Multisig Notary requested

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

100TiB

galen-mcandrew commented 2 years ago

@parkan Here is the new large dataset application issue, per the new LDN process.

large-datacap-requests[bot] commented 2 years ago

**Multisig created and sent to RKH f01322605

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Multisig Notary address

f01322605

Client address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

DataCap allocation requested

50TiB

cryptowhizzard commented 2 years ago

Looks good to me.

Reiers commented 2 years ago

+1 seems good to me too, but I cant find the request on plus.fil dashboard.

galen-mcandrew commented 2 years ago

@Reiers & @cryptowhizzard thanks for flagging! Not sure why it wasn't showing up in the app. I'm seeing it now, can you try again? Org address: f01322605

dannyob commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedfdxlezwbyddmhojnyctgby4w7hoegkegeuhmwrbswf6apli36hw

Address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Datacap Allocated

50TiB

Signer Address

f1k6wwevxvp466ybil7y2scqlhtnrz5atjkkyvm4a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedfdxlezwbyddmhojnyctgby4w7hoegkegeuhmwrbswf6apli36hw

MegTei commented 2 years ago

Hmm, 1st error was Block store failed. Now this. Process Q: once an approved client do subsequent allocations continue to require 2 notaries?

dkkapur commented 2 years ago

@MegTei I think this actually went through, that seems to just be a transient error on the site: https://filfox.info/en/message/bafy2bzacea4i245gwngvcvgk2uiwyjhjgjjdsmtq7bxqjvl4a24ombs7q7fyy

galen-mcandrew commented 2 years ago

Yes, all subsequent allocations still require two notary signatures, the threshold on the multisig is 2. While not a hard-coded setting right now, the current process calls for two different notaries to sign the subsequent allocation. You will be able to see previous signors in the stats bot comment just above the subsequent allocation in the github issue.

fabriziogianni7 commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacea4i245gwngvcvgk2uiwyjhjgjjdsmtq7bxqjvl4a24ombs7q7fyy

Address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Datacap Allocated

50TiB

Signer Address

f1ystxl2ootvpirpa7ebgwl7vlhwkbx2r4zjxwe5i

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea4i245gwngvcvgk2uiwyjhjgjjdsmtq7bxqjvl4a24ombs7q7fyy

fabriziogianni7 commented 2 years ago

adding the comment for the second approval to make it readable from the bot. cc @MegTei @dkkapur @galen-mcandrew

parkan commented 2 years ago

An update from the Internet Archive:

We're happy to let the Notaries and the Filecoin community know that we have finally managed to scale up our transfer operations to production levels, and are currently transmitting over 1TiB/day so service providers, with an eye on getting this up to 10TiB/day or more. It took us a while to get to this point due to unique characteristics of the Archive infrastructure and the way they interact with design assumptions of Filecoin tooling; a huge thanks to the PL team for supporting us on our journey and to the notaries and the community for their patience.

The currently remaining DataCap for our wallet now stands at 85TiB and is being used up rapidly, likely running out in the next couple of weeks. What is the process for receiving the subsequent allocations? @galen-mcandrew please let me know what actions are needed from our side. Thank you!

parkan commented 2 years ago

We've crossed 69TiB remaining now, am I correct in understanding that at 75TiB remaining the next allocation process should have kicked off?

parkan commented 2 years ago

Hello Notaries! We have now crossed below 25TiB (we've slowed down for a bit to address some issues with tooling), I believe this should trigger the next DataCap tranche. Please let me know what we need to do to proceed.

dkkapur commented 2 years ago

@parkan FYI its actually at 25% remaining, not 25 TiB remaining. Your last allocation was 50 TiB, so the bot will auto trigger at 12.5 TiB.

What's your current rate of daily onboarding? If this is < 2d of runway, IMO we should request notaries to sign sooner.

parkan commented 2 years ago

@parkan FYI its actually at 25% remaining, not 25 TiB remaining. Your last allocation was 50 TiB, so the bot will auto trigger at 12.5 TiB.

What's your current rate of daily onboarding? If this is < 2d of runway, IMO we should request notaries to sign sooner.

ah my mistake, I thought my initial batch was 100TiB this 25TiB=25%

we have some data staged for large offline deals so it would be great to have this signed soon

parkan commented 2 years ago

we're currently paused on ingestion since the 25TiB margin is too low to do the bulk offline deals as well as the ongoing online deals, could you please advise how we can get this rolling?

dkkapur commented 2 years ago

@parkan that's great feedback. thank you. what is a realistic weekly onboarding rate for you moving forward? I think we should push through the next allocation sooner and adjust the rate as well.

parkan commented 2 years ago

@dkkapur we're currently targeting around 100TiB/week, possibly more when we onboard more SPs

dkkapur commented 2 years ago

Sounds good. The next allocation is expected to be for 100 TiB. I assume that should be good to get this rolling?

Notaries - I'm triggering the next allocation sooner but about 10 TiB. please feel free to ask any questions. Also migrating this to be LDN v3 based as a public dataset.

dkkapur commented 2 years ago

DataCap Allocation requested

Multisig Notary address

f01858410

Client address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

DataCap allocation requested

100TiB

psh0691 commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceatbxp3jmxfffzhr55mwgggonthpssma2wztblf6zjdcupz27cig6

Address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Datacap Allocated

100.00TiB

Signer Address

f1qdko4jg25vo35qmyvcrw4ak4fmuu3f5rif2kc7i

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceatbxp3jmxfffzhr55mwgggonthpssma2wztblf6zjdcupz27cig6

neogeweb3 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedh6tobo6y67vo6yvd3tachiqr5yh2ydglwxategtinjxvkptn3xu

Address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Datacap Allocated

100.00TiB

Signer Address

f13k5zr6ovc2gjmg3lvd43ladbydhovpylcvbflpa

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedh6tobo6y67vo6yvd3tachiqr5yh2ydglwxategtinjxvkptn3xu

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f01858410

Client address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

DataCap allocation requested

200TiB

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Last two approvers

neogeweb3 & psh0691

Rule to calculate the allocation request amount

200% of weekly dc amount requested

DataCap allocation requested

200TiB

Total DataCap granted for client so far

96GiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

4.99PiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
4	3	100TiB	57.14	13.78TiB

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecm3ozfwyt2cgjbk7x4rdgpbjn6k5hkkhpksj3iz46zeo6rlc4qig

Address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Datacap Allocated

200.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecm3ozfwyt2cgjbk7x4rdgpbjn6k5hkkhpksj3iz46zeo6rlc4qig

Alex11801 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacea43yzzirtqm6fd6y5uuoaukkz63pobeoyylwwuu6ybyat3xka4kk

Address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Datacap Allocated

200.00TiB

Signer Address

f1hhippi64yiyhpjdtbidfyzma6irc2nuav7mrwmi

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea43yzzirtqm6fd6y5uuoaukkz63pobeoyylwwuu6ybyat3xka4kk

filplus-checker commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: The Internet Archive
Client: f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Storage provider should not exceed 25% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ f01611097 has sealed 32.00% of total datacap.

⚠️ f020378 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01611097	San Clemente, California, US	73.78 TiB	32.00%	71.66 TiB	2.87%
f01910202	Philadelphia, Pennsylvania, US	44.68 TiB	19.38%	44.25 TiB	0.96%
f01873432	Las Vegas, Nevada, US	27.92 TiB	12.11%	26.62 TiB	4.67%
f01904630	Las Vegas, Nevada, US	23.47 TiB	10.18%	20.62 TiB	12.16%
f01882184	Singapore, Singapore, SG	18.81 TiB	8.16%	18.81 TiB	0.00%
f01826669	Philadelphia, Pennsylvania, US	18.29 TiB	7.93%	18.29 TiB	0.00%
f01851683`new`	Las Vegas, Nevada, US	11.52 TiB	4.99%	11.52 TiB	0.00%
f01883179`new`	Philadelphia, Pennsylvania, US	7.69 TiB	3.33%	7.69 TiB	0.00%
f01606675	Montréal, Quebec, CA	3.30 TiB	1.43%	3.30 TiB	0.00%
f066596	San Diego, California, US	274.00 GiB	0.12%	274.00 GiB	0.00%
f01091840	Montréal, Quebec, CA	176.00 GiB	0.07%	160.00 GiB	9.09%
f01199442	Heerhugowaard, North Holland, NL	146.00 GiB	0.06%	130.00 GiB	10.96%
f02576	Copenhagen, Capital Region, DK	96.00 GiB	0.04%	96.00 GiB	0.00%
f0157535	Montréal, Quebec, CA	80.00 GiB	0.03%	80.00 GiB	0.00%
f0104671	Kawasaki, Kanagawa, JP	64.00 GiB	0.03%	64.00 GiB	0.00%
f019104	Montréal, Quebec, CA	53.00 GiB	0.02%	53.00 GiB	0.00%
f09848	Rancho Santa Margarita, California, US	48.00 GiB	0.02%	48.00 GiB	0.00%
f0165400	Montréal, Quebec, CA	48.00 GiB	0.02%	48.00 GiB	0.00%
f01207045	Heerhugowaard, North Holland, NL	32.00 GiB	0.01%	32.00 GiB	0.00%
f058369	Boston, Massachusetts, US	32.00 GiB	0.01%	32.00 GiB	0.00%
f010088	Everett, Washington, US	18.00 GiB	0.01%	18.00 GiB	0.00%
f0694396	Birmingham, England, GB	17.00 GiB	0.01%	17.00 GiB	0.00%
f019551	Birmingham, England, GB	16.00 GiB	0.01%	16.00 GiB	0.00%
f030379	Seoul, Seoul, KR	16.00 GiB	0.01%	16.00 GiB	0.00%
f010446	Zaventem, Flanders, BE	16.00 GiB	0.01%	16.00 GiB	0.00%
f01199430	Heerhugowaard, North Holland, NL	3.00 GiB	0.00%	3.00 GiB	0.00%
f024184	Seoul, Seoul, KR	2.00 GiB	0.00%	2.00 GiB	0.00%
f020378	Unknown	2.00 GiB	0.00%	2.00 GiB	0.00%
f01345523	Antwerpen, Flanders, BE	2.00 GiB	0.00%	2.00 GiB	0.00%
f01784458	Oslo, Oslo, NO	1.00 GiB	0.00%	1.00 GiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

No more than 25% of unique data are stored with less than 4 providers.

⚠️ 83.14% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
26.40 TiB	26.41 TiB	1	11.45%
28.93 TiB	58.10 TiB	2	25.20%
35.01 TiB	107.20 TiB	3	46.49%
8.64 TiB	38.87 TiB	4	16.86%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Verifier
f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy	`Glif auto verified`	208.00 GiB	1	Jonathan Schwartz
f3wkp4blevjsrtbc6vwgjf2sedzjwsqmj3wsh4uex bp4k7dggs72kbvuv7xivsnz7cnmfazpmqp3qmchmz ms6a	Unknown	208.00 GiB	1	Unknown
f3u5dehxxe2uvehitioxhwjp27wpv72hsnuqhtz6s ce2wzqv2skhguivnsvwbkwgczcc5x4qf6eeao34te jqdq	`Glif auto verified`	16.00 GiB	1	Jonathan Schwartz

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: The Internet Archive
Client: f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Since this is the 4th allocation, the following restrictions have been relaxed:

Storage provider should not exceed 50% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ f020378 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01611097	San Clemente, California, US	73.78 TiB	32.00%	71.66 TiB	2.87%
f01910202	Philadelphia, Pennsylvania, US	44.68 TiB	19.38%	44.25 TiB	0.96%
f01873432	Las Vegas, Nevada, US	27.92 TiB	12.11%	26.62 TiB	4.67%
f01904630	Las Vegas, Nevada, US	23.47 TiB	10.18%	20.62 TiB	12.16%
f01882184	Singapore, Singapore, SG	18.81 TiB	8.16%	18.81 TiB	0.00%
f01826669	Philadelphia, Pennsylvania, US	18.29 TiB	7.93%	18.29 TiB	0.00%
f01851683`new`	Las Vegas, Nevada, US	11.52 TiB	4.99%	11.52 TiB	0.00%
f01883179`new`	Philadelphia, Pennsylvania, US	7.69 TiB	3.33%	7.69 TiB	0.00%
f01606675	Montréal, Quebec, CA	3.30 TiB	1.43%	3.30 TiB	0.00%
f066596	San Diego, California, US	274.00 GiB	0.12%	274.00 GiB	0.00%
f01091840	Montréal, Quebec, CA	176.00 GiB	0.07%	160.00 GiB	9.09%
f01199442	Heerhugowaard, North Holland, NL	146.00 GiB	0.06%	130.00 GiB	10.96%
f02576	Copenhagen, Capital Region, DK	96.00 GiB	0.04%	96.00 GiB	0.00%
f0157535	Montréal, Quebec, CA	80.00 GiB	0.03%	80.00 GiB	0.00%
f0104671	Kawasaki, Kanagawa, JP	64.00 GiB	0.03%	64.00 GiB	0.00%
f019104	Montréal, Quebec, CA	53.00 GiB	0.02%	53.00 GiB	0.00%
f09848	Rancho Santa Margarita, California, US	48.00 GiB	0.02%	48.00 GiB	0.00%
f0165400	Montréal, Quebec, CA	48.00 GiB	0.02%	48.00 GiB	0.00%
f01207045	Heerhugowaard, North Holland, NL	32.00 GiB	0.01%	32.00 GiB	0.00%
f058369	Boston, Massachusetts, US	32.00 GiB	0.01%	32.00 GiB	0.00%
f010088	Everett, Washington, US	18.00 GiB	0.01%	18.00 GiB	0.00%
f0694396	Birmingham, England, GB	17.00 GiB	0.01%	17.00 GiB	0.00%
f019551	Birmingham, England, GB	16.00 GiB	0.01%	16.00 GiB	0.00%
f030379	Seoul, Seoul, KR	16.00 GiB	0.01%	16.00 GiB	0.00%
f010446	Zaventem, Flanders, BE	16.00 GiB	0.01%	16.00 GiB	0.00%
f01199430	Heerhugowaard, North Holland, NL	3.00 GiB	0.00%	3.00 GiB	0.00%
f024184	Seoul, Seoul, KR	2.00 GiB	0.00%	2.00 GiB	0.00%
f020378	Unknown	2.00 GiB	0.00%	2.00 GiB	0.00%
f01345523	Antwerpen, Flanders, BE	2.00 GiB	0.00%	2.00 GiB	0.00%
f01784458	Oslo, Oslo, NO	1.00 GiB	0.00%	1.00 GiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

Since this is the 4th allocation, the following restrictions have been relaxed:

No more than 50% of unique data are stored with less than 4 providers.

⚠️ 83.14% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
26.40 TiB	26.41 TiB	1	11.45%
28.93 TiB	58.10 TiB	2	25.20%
35.01 TiB	107.20 TiB	3	46.49%
8.64 TiB	38.87 TiB	4	16.86%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Verifier
f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy	`Glif auto verified`	208.00 GiB	1	Jonathan Schwartz
f3wkp4blevjsrtbc6vwgjf2sedzjwsqmj3wsh4uex bp4k7dggs72kbvuv7xivsnz7cnmfazpmqp3qmchmz ms6a	Unknown	208.00 GiB	1	Unknown
f3u5dehxxe2uvehitioxhwjp27wpv72hsnuqhtz6s ce2wzqv2skhguivnsvwbkwgczcc5x4qf6eeao34te jqdq	`Glif auto verified`	16.00 GiB	1	Jonathan Schwartz

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: The Internet Archive
Client: f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Since this is the 4th allocation, the following restrictions have been relaxed:

Storage provider should not exceed 50% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ f020378 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01611097	San Clemente, California, US	73.78 TiB	32.00%	71.66 TiB	2.87%
f01910202	Philadelphia, Pennsylvania, US	44.68 TiB	19.38%	44.25 TiB	0.96%
f01873432	Las Vegas, Nevada, US	27.92 TiB	12.11%	26.62 TiB	4.67%
f01904630	Las Vegas, Nevada, US	23.47 TiB	10.18%	20.62 TiB	12.16%
f01882184	Singapore, Singapore, SG	18.81 TiB	8.16%	18.81 TiB	0.00%
f01826669	Philadelphia, Pennsylvania, US	18.29 TiB	7.93%	18.29 TiB	0.00%
f01851683`new`	Las Vegas, Nevada, US	11.52 TiB	4.99%	11.52 TiB	0.00%
f01883179`new`	Philadelphia, Pennsylvania, US	7.69 TiB	3.33%	7.69 TiB	0.00%
f01606675	Montréal, Quebec, CA	3.30 TiB	1.43%	3.30 TiB	0.00%
f066596	San Diego, California, US	274.00 GiB	0.12%	274.00 GiB	0.00%
f01091840	Montréal, Quebec, CA	176.00 GiB	0.07%	160.00 GiB	9.09%
f01199442	Heerhugowaard, North Holland, NL	146.00 GiB	0.06%	130.00 GiB	10.96%
f02576	Copenhagen, Capital Region, DK	96.00 GiB	0.04%	96.00 GiB	0.00%
f0157535	Montréal, Quebec, CA	80.00 GiB	0.03%	80.00 GiB	0.00%
f0104671	Kawasaki, Kanagawa, JP	64.00 GiB	0.03%	64.00 GiB	0.00%
f019104	Montréal, Quebec, CA	53.00 GiB	0.02%	53.00 GiB	0.00%
f09848	Rancho Santa Margarita, California, US	48.00 GiB	0.02%	48.00 GiB	0.00%
f0165400	Montréal, Quebec, CA	48.00 GiB	0.02%	48.00 GiB	0.00%
f01207045	Heerhugowaard, North Holland, NL	32.00 GiB	0.01%	32.00 GiB	0.00%
f058369	Boston, Massachusetts, US	32.00 GiB	0.01%	32.00 GiB	0.00%
f010088	Everett, Washington, US	18.00 GiB	0.01%	18.00 GiB	0.00%
f0694396	Birmingham, England, GB	17.00 GiB	0.01%	17.00 GiB	0.00%
f019551	Birmingham, England, GB	16.00 GiB	0.01%	16.00 GiB	0.00%
f030379	Seoul, Seoul, KR	16.00 GiB	0.01%	16.00 GiB	0.00%
f010446	Zaventem, Flanders, BE	16.00 GiB	0.01%	16.00 GiB	0.00%
f01199430	Heerhugowaard, North Holland, NL	3.00 GiB	0.00%	3.00 GiB	0.00%
f024184	Seoul, Seoul, KR	2.00 GiB	0.00%	2.00 GiB	0.00%
f020378	Unknown	2.00 GiB	0.00%	2.00 GiB	0.00%
f01345523	Antwerpen, Flanders, BE	2.00 GiB	0.00%	2.00 GiB	0.00%
f01784458	Oslo, Oslo, NO	1.00 GiB	0.00%	1.00 GiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

Since this is the 4th allocation, the following restrictions have been relaxed:

No more than 50% of unique data are stored with less than 4 providers.

⚠️ 83.14% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
26.40 TiB	26.41 TiB	1	11.45%
28.93 TiB	58.10 TiB	2	25.20%
35.01 TiB	107.20 TiB	3	46.49%
8.64 TiB	38.87 TiB	4	16.86%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Verifier
f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy	`Glif auto verified`	208.00 GiB	1	Jonathan Schwartz
f3wkp4blevjsrtbc6vwgjf2sedzjwsqmj3wsh4uex bp4k7dggs72kbvuv7xivsnz7cnmfazpmqp3qmchmz ms6a	Unknown	208.00 GiB	1	Unknown
f3u5dehxxe2uvehitioxhwjp27wpv72hsnuqhtz6s ce2wzqv2skhguivnsvwbkwgczcc5x4qf6eeao34te jqdq	`Glif auto verified`	16.00 GiB	1	Jonathan Schwartz

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: The Internet Archive
Client: f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Since this is the 4th allocation, the following restrictions have been relaxed:

Storage provider should not exceed 50% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ f020378 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01611097	San Clemente, California, US	73.78 TiB	32.00%	71.66 TiB	2.87%
f01910202	Philadelphia, Pennsylvania, US	44.68 TiB	19.38%	44.25 TiB	0.96%
f01873432	Las Vegas, Nevada, US	27.92 TiB	12.11%	26.62 TiB	4.67%
f01904630	Las Vegas, Nevada, US	23.47 TiB	10.18%	20.62 TiB	12.16%
f01882184	Singapore, Singapore, SG	18.81 TiB	8.16%	18.81 TiB	0.00%
f01826669	Philadelphia, Pennsylvania, US	18.29 TiB	7.93%	18.29 TiB	0.00%
f01851683`new`	Las Vegas, Nevada, US	11.52 TiB	4.99%	11.52 TiB	0.00%
f01883179`new`	Philadelphia, Pennsylvania, US	7.69 TiB	3.33%	7.69 TiB	0.00%
f01606675	Montréal, Quebec, CA	3.30 TiB	1.43%	3.30 TiB	0.00%
f066596	San Diego, California, US	274.00 GiB	0.12%	274.00 GiB	0.00%
f01091840	Montréal, Quebec, CA	176.00 GiB	0.07%	160.00 GiB	9.09%
f01199442	Heerhugowaard, North Holland, NL	146.00 GiB	0.06%	130.00 GiB	10.96%
f02576	Copenhagen, Capital Region, DK	96.00 GiB	0.04%	96.00 GiB	0.00%
f0157535	Montréal, Quebec, CA	80.00 GiB	0.03%	80.00 GiB	0.00%
f0104671	Kawasaki, Kanagawa, JP	64.00 GiB	0.03%	64.00 GiB	0.00%
f019104	Montréal, Quebec, CA	53.00 GiB	0.02%	53.00 GiB	0.00%
f09848	Rancho Santa Margarita, California, US	48.00 GiB	0.02%	48.00 GiB	0.00%
f0165400	Montréal, Quebec, CA	48.00 GiB	0.02%	48.00 GiB	0.00%
f01207045	Heerhugowaard, North Holland, NL	32.00 GiB	0.01%	32.00 GiB	0.00%
f058369	Boston, Massachusetts, US	32.00 GiB	0.01%	32.00 GiB	0.00%
f010088	Everett, Washington, US	18.00 GiB	0.01%	18.00 GiB	0.00%
f0694396	Birmingham, England, GB	17.00 GiB	0.01%	17.00 GiB	0.00%
f019551	Birmingham, England, GB	16.00 GiB	0.01%	16.00 GiB	0.00%
f030379	Seoul, Seoul, KR	16.00 GiB	0.01%	16.00 GiB	0.00%
f010446	Zaventem, Flanders, BE	16.00 GiB	0.01%	16.00 GiB	0.00%
f01199430	Heerhugowaard, North Holland, NL	3.00 GiB	0.00%	3.00 GiB	0.00%
f024184	Seoul, Seoul, KR	2.00 GiB	0.00%	2.00 GiB	0.00%
f020378	Unknown	2.00 GiB	0.00%	2.00 GiB	0.00%
f01345523	Antwerpen, Flanders, BE	2.00 GiB	0.00%	2.00 GiB	0.00%
f01784458	Oslo, Oslo, NO	1.00 GiB	0.00%	1.00 GiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

Since this is the 4th allocation, the following restrictions have been relaxed:

No more than 50% of unique data are stored with less than 4 providers.

⚠️ 83.14% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
26.40 TiB	26.41 TiB	1	11.45%
28.93 TiB	58.10 TiB	2	25.20%
35.01 TiB	107.20 TiB	3	46.49%
8.64 TiB	38.87 TiB	4	16.86%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Verifier
f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy	`Glif auto verified`	208.00 GiB	1	Jonathan Schwartz
f3wkp4blevjsrtbc6vwgjf2sedzjwsqmj3wsh4uex bp4k7dggs72kbvuv7xivsnz7cnmfazpmqp3qmchmz ms6a	Unknown	208.00 GiB	1	Unknown
f3u5dehxxe2uvehitioxhwjp27wpv72hsnuqhtz6s ce2wzqv2skhguivnsvwbkwgczcc5x4qf6eeao34te jqdq	`Glif auto verified`	16.00 GiB	1	Jonathan Schwartz

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: The Internet Archive
Client: f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Since this is the 4th allocation, the following restrictions have been relaxed:

Storage provider should not exceed 50% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ f020378 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01611097	San Clemente, California, US	73.78 TiB	32.00%	71.66 TiB	2.87%
f01910202	Philadelphia, Pennsylvania, US	44.68 TiB	19.38%	44.25 TiB	0.96%
f01873432	Las Vegas, Nevada, US	27.92 TiB	12.11%	26.62 TiB	4.67%
f01904630	Las Vegas, Nevada, US	23.47 TiB	10.18%	20.62 TiB	12.16%
f01882184	Singapore, Singapore, SG	18.81 TiB	8.16%	18.81 TiB	0.00%
f01826669	Philadelphia, Pennsylvania, US	18.29 TiB	7.93%	18.29 TiB	0.00%
f01851683`new`	Las Vegas, Nevada, US	11.52 TiB	4.99%	11.52 TiB	0.00%
f01883179`new`	Philadelphia, Pennsylvania, US	7.69 TiB	3.33%	7.69 TiB	0.00%
f01606675	Montréal, Quebec, CA	3.30 TiB	1.43%	3.30 TiB	0.00%
f066596	San Diego, California, US	274.00 GiB	0.12%	274.00 GiB	0.00%
f01091840	Montréal, Quebec, CA	176.00 GiB	0.07%	160.00 GiB	9.09%
f01199442	Heerhugowaard, North Holland, NL	146.00 GiB	0.06%	130.00 GiB	10.96%
f02576	Copenhagen, Capital Region, DK	96.00 GiB	0.04%	96.00 GiB	0.00%
f0157535	Montréal, Quebec, CA	80.00 GiB	0.03%	80.00 GiB	0.00%
f0104671	Kawasaki, Kanagawa, JP	64.00 GiB	0.03%	64.00 GiB	0.00%
f019104	Montréal, Quebec, CA	53.00 GiB	0.02%	53.00 GiB	0.00%
f09848	Rancho Santa Margarita, California, US	48.00 GiB	0.02%	48.00 GiB	0.00%
f0165400	Montréal, Quebec, CA	48.00 GiB	0.02%	48.00 GiB	0.00%
f01207045	Heerhugowaard, North Holland, NL	32.00 GiB	0.01%	32.00 GiB	0.00%
f058369	Boston, Massachusetts, US	32.00 GiB	0.01%	32.00 GiB	0.00%
f010088	Everett, Washington, US	18.00 GiB	0.01%	18.00 GiB	0.00%
f0694396	Birmingham, England, GB	17.00 GiB	0.01%	17.00 GiB	0.00%
f019551	Birmingham, England, GB	16.00 GiB	0.01%	16.00 GiB	0.00%
f030379	Seoul, Seoul, KR	16.00 GiB	0.01%	16.00 GiB	0.00%
f010446	Zaventem, Flanders, BE	16.00 GiB	0.01%	16.00 GiB	0.00%
f01199430	Heerhugowaard, North Holland, NL	3.00 GiB	0.00%	3.00 GiB	0.00%
f024184	Seoul, Seoul, KR	2.00 GiB	0.00%	2.00 GiB	0.00%
f020378	Unknown	2.00 GiB	0.00%	2.00 GiB	0.00%
f01345523	Antwerpen, Flanders, BE	2.00 GiB	0.00%	2.00 GiB	0.00%
f01784458	Oslo, Oslo, NO	1.00 GiB	0.00%	1.00 GiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

Since this is the 4th allocation, the following restrictions have been relaxed:

No more than 50% of unique data are stored with less than 4 providers.

⚠️ 83.14% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
26.40 TiB	26.41 TiB	1	11.45%
28.93 TiB	58.10 TiB	2	25.20%
35.01 TiB	107.20 TiB	3	46.49%
8.64 TiB	38.87 TiB	4	16.86%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Verifier
f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy	`Glif auto verified`	208.00 GiB	1	Jonathan Schwartz
f3wkp4blevjsrtbc6vwgjf2sedzjwsqmj3wsh4uex bp4k7dggs72kbvuv7xivsnz7cnmfazpmqp3qmchmz ms6a	Unknown	208.00 GiB	1	Unknown
f3u5dehxxe2uvehitioxhwjp27wpv72hsnuqhtz6s ce2wzqv2skhguivnsvwbkwgczcc5x4qf6eeao34te jqdq	`Glif auto verified`	16.00 GiB	1	Jonathan Schwartz

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: The Internet Archive
Client: f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Since this is the 4th allocation, the following restrictions have been relaxed:

Storage provider should not exceed 50% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ f020378 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01611097	San Clemente, California, US	73.78 TiB	32.00%	71.66 TiB	2.87%
f01910202	Philadelphia, Pennsylvania, US	44.68 TiB	19.38%	44.25 TiB	0.96%
f01873432	Las Vegas, Nevada, US	27.92 TiB	12.11%	26.62 TiB	4.67%
f01904630	Las Vegas, Nevada, US	23.47 TiB	10.18%	20.62 TiB	12.16%
f01882184	Singapore, Singapore, SG	18.81 TiB	8.16%	18.81 TiB	0.00%
f01826669	Philadelphia, Pennsylvania, US	18.29 TiB	7.93%	18.29 TiB	0.00%
f01851683`new`	Las Vegas, Nevada, US	11.52 TiB	4.99%	11.52 TiB	0.00%
f01883179`new`	Philadelphia, Pennsylvania, US	7.69 TiB	3.33%	7.69 TiB	0.00%
f01606675	Montréal, Quebec, CA	3.30 TiB	1.43%	3.30 TiB	0.00%
f066596	San Diego, California, US	274.00 GiB	0.12%	274.00 GiB	0.00%
f01091840	Montréal, Quebec, CA	176.00 GiB	0.07%	160.00 GiB	9.09%
f01199442	Heerhugowaard, North Holland, NL	146.00 GiB	0.06%	130.00 GiB	10.96%
f02576	Copenhagen, Capital Region, DK	96.00 GiB	0.04%	96.00 GiB	0.00%
f0157535	Montréal, Quebec, CA	80.00 GiB	0.03%	80.00 GiB	0.00%
f0104671	Kawasaki, Kanagawa, JP	64.00 GiB	0.03%	64.00 GiB	0.00%
f019104	Montréal, Quebec, CA	53.00 GiB	0.02%	53.00 GiB	0.00%
f09848	Rancho Santa Margarita, California, US	48.00 GiB	0.02%	48.00 GiB	0.00%
f0165400	Montréal, Quebec, CA	48.00 GiB	0.02%	48.00 GiB	0.00%
f01207045	Heerhugowaard, North Holland, NL	32.00 GiB	0.01%	32.00 GiB	0.00%
f058369	Boston, Massachusetts, US	32.00 GiB	0.01%	32.00 GiB	0.00%
f010088	Everett, Washington, US	18.00 GiB	0.01%	18.00 GiB	0.00%
f0694396	Birmingham, England, GB	17.00 GiB	0.01%	17.00 GiB	0.00%
f019551	Birmingham, England, GB	16.00 GiB	0.01%	16.00 GiB	0.00%
f030379	Seoul, Seoul, KR	16.00 GiB	0.01%	16.00 GiB	0.00%
f010446	Zaventem, Flanders, BE	16.00 GiB	0.01%	16.00 GiB	0.00%
f01199430	Heerhugowaard, North Holland, NL	3.00 GiB	0.00%	3.00 GiB	0.00%
f024184	Seoul, Seoul, KR	2.00 GiB	0.00%	2.00 GiB	0.00%
f020378	Unknown	2.00 GiB	0.00%	2.00 GiB	0.00%
f01345523	Antwerpen, Flanders, BE	2.00 GiB	0.00%	2.00 GiB	0.00%
f01784458	Oslo, Oslo, NO	1.00 GiB	0.00%	1.00 GiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

Since this is the 4th allocation, the following restrictions have been relaxed:

No more than 50% of unique data are stored with less than 4 providers.

⚠️ 83.14% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
26.40 TiB	26.41 TiB	1	11.45%
28.93 TiB	58.10 TiB	2	25.20%
35.01 TiB	107.20 TiB	3	46.49%
8.64 TiB	38.87 TiB	4	16.86%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Verifier
f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy	`Glif auto verified`	208.00 GiB	1	Jonathan Schwartz
f3wkp4blevjsrtbc6vwgjf2sedzjwsqmj3wsh4uex bp4k7dggs72kbvuv7xivsnz7cnmfazpmqp3qmchmz ms6a	Unknown	208.00 GiB	1	Unknown
f3u5dehxxe2uvehitioxhwjp27wpv72hsnuqhtz6s ce2wzqv2skhguivnsvwbkwgczcc5x4qf6eeao34te jqdq	`Glif auto verified`	16.00 GiB	1	Jonathan Schwartz

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: The Internet Archive
Client: f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Since this is the 4th allocation, the following restrictions have been relaxed:

Storage provider should not exceed 50% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ f020378 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01611097	San Clemente, California, US	73.78 TiB	32.00%	71.66 TiB	2.87%
f01910202	Philadelphia, Pennsylvania, US	44.68 TiB	19.38%	44.25 TiB	0.96%
f01873432	Las Vegas, Nevada, US	27.92 TiB	12.11%	26.62 TiB	4.67%
f01904630	Las Vegas, Nevada, US	23.47 TiB	10.18%	20.62 TiB	12.16%
f01882184	Singapore, Singapore, SG	18.81 TiB	8.16%	18.81 TiB	0.00%
f01826669	Philadelphia, Pennsylvania, US	18.29 TiB	7.93%	18.29 TiB	0.00%
f01851683`new`	Las Vegas, Nevada, US	11.52 TiB	4.99%	11.52 TiB	0.00%
f01883179`new`	Philadelphia, Pennsylvania, US	7.69 TiB	3.33%	7.69 TiB	0.00%
f01606675	Montréal, Quebec, CA	3.30 TiB	1.43%	3.30 TiB	0.00%
f066596	San Diego, California, US	274.00 GiB	0.12%	274.00 GiB	0.00%
f01091840	Montréal, Quebec, CA	176.00 GiB	0.07%	160.00 GiB	9.09%
f01199442	Heerhugowaard, North Holland, NL	146.00 GiB	0.06%	130.00 GiB	10.96%
f02576	Copenhagen, Capital Region, DK	96.00 GiB	0.04%	96.00 GiB	0.00%
f0157535	Montréal, Quebec, CA	80.00 GiB	0.03%	80.00 GiB	0.00%
f0104671	Kawasaki, Kanagawa, JP	64.00 GiB	0.03%	64.00 GiB	0.00%
f019104	Montréal, Quebec, CA	53.00 GiB	0.02%	53.00 GiB	0.00%
f09848	Rancho Santa Margarita, California, US	48.00 GiB	0.02%	48.00 GiB	0.00%
f0165400	Montréal, Quebec, CA	48.00 GiB	0.02%	48.00 GiB	0.00%
f01207045	Heerhugowaard, North Holland, NL	32.00 GiB	0.01%	32.00 GiB	0.00%
f058369	Boston, Massachusetts, US	32.00 GiB	0.01%	32.00 GiB	0.00%
f010088	Everett, Washington, US	18.00 GiB	0.01%	18.00 GiB	0.00%
f0694396	Birmingham, England, GB	17.00 GiB	0.01%	17.00 GiB	0.00%
f019551	Birmingham, England, GB	16.00 GiB	0.01%	16.00 GiB	0.00%
f030379	Seoul, Seoul, KR	16.00 GiB	0.01%	16.00 GiB	0.00%
f010446	Zaventem, Flanders, BE	16.00 GiB	0.01%	16.00 GiB	0.00%
f01199430	Heerhugowaard, North Holland, NL	3.00 GiB	0.00%	3.00 GiB	0.00%
f024184	Seoul, Seoul, KR	2.00 GiB	0.00%	2.00 GiB	0.00%
f020378	Unknown	2.00 GiB	0.00%	2.00 GiB	0.00%
f01345523	Antwerpen, Flanders, BE	2.00 GiB	0.00%	2.00 GiB	0.00%
f01784458	Oslo, Oslo, NO	1.00 GiB	0.00%	1.00 GiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

Since this is the 4th allocation, the following restrictions have been relaxed:

No more than 50% of unique data are stored with less than 4 providers.

⚠️ 83.14% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
26.40 TiB	26.41 TiB	1	11.45%
28.93 TiB	58.10 TiB	2	25.20%
35.01 TiB	107.20 TiB	3	46.49%
8.64 TiB	38.87 TiB	4	16.86%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Verifier
f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy	`Glif auto verified`	208.00 GiB	1	Jonathan Schwartz
f3wkp4blevjsrtbc6vwgjf2sedzjwsqmj3wsh4uex bp4k7dggs72kbvuv7xivsnz7cnmfazpmqp3qmchmz ms6a	Unknown	208.00 GiB	1	Unknown
f3u5dehxxe2uvehitioxhwjp27wpv72hsnuqhtz6s ce2wzqv2skhguivnsvwbkwgczcc5x4qf6eeao34te jqdq	`Glif auto verified`	16.00 GiB	1	Jonathan Schwartz

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: The Internet Archive
Client: f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Since this is the 4th allocation, the following restrictions have been relaxed:

Storage provider should not exceed 50% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ f020378 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01611097	San Clemente, California, US	73.78 TiB	32.00%	71.66 TiB	2.87%
f01910202	Philadelphia, Pennsylvania, US	44.68 TiB	19.38%	44.25 TiB	0.96%
f01873432	Las Vegas, Nevada, US	27.92 TiB	12.11%	26.62 TiB	4.67%
f01904630	Las Vegas, Nevada, US	23.47 TiB	10.18%	20.62 TiB	12.16%
f01882184	Singapore, Singapore, SG	18.81 TiB	8.16%	18.81 TiB	0.00%
f01826669	Philadelphia, Pennsylvania, US	18.29 TiB	7.93%	18.29 TiB	0.00%
f01851683`new`	Las Vegas, Nevada, US	11.52 TiB	4.99%	11.52 TiB	0.00%
f01883179`new`	Philadelphia, Pennsylvania, US	7.69 TiB	3.33%	7.69 TiB	0.00%
f01606675	Montréal, Quebec, CA	3.30 TiB	1.43%	3.30 TiB	0.00%
f066596	San Diego, California, US	274.00 GiB	0.12%	274.00 GiB	0.00%
f01091840	Montréal, Quebec, CA	176.00 GiB	0.07%	160.00 GiB	9.09%
f01199442	Heerhugowaard, North Holland, NL	146.00 GiB	0.06%	130.00 GiB	10.96%
f02576	Copenhagen, Capital Region, DK	96.00 GiB	0.04%	96.00 GiB	0.00%
f0157535	Montréal, Quebec, CA	80.00 GiB	0.03%	80.00 GiB	0.00%
f0104671	Kawasaki, Kanagawa, JP	64.00 GiB	0.03%	64.00 GiB	0.00%
f019104	Montréal, Quebec, CA	53.00 GiB	0.02%	53.00 GiB	0.00%
f09848	Rancho Santa Margarita, California, US	48.00 GiB	0.02%	48.00 GiB	0.00%
f0165400	Montréal, Quebec, CA	48.00 GiB	0.02%	48.00 GiB	0.00%
f01207045	Heerhugowaard, North Holland, NL	32.00 GiB	0.01%	32.00 GiB	0.00%
f058369	Boston, Massachusetts, US	32.00 GiB	0.01%	32.00 GiB	0.00%
f010088	Everett, Washington, US	18.00 GiB	0.01%	18.00 GiB	0.00%
f0694396	Birmingham, England, GB	17.00 GiB	0.01%	17.00 GiB	0.00%
f019551	Birmingham, England, GB	16.00 GiB	0.01%	16.00 GiB	0.00%
f030379	Seoul, Seoul, KR	16.00 GiB	0.01%	16.00 GiB	0.00%
f010446	Zaventem, Flanders, BE	16.00 GiB	0.01%	16.00 GiB	0.00%
f01199430	Heerhugowaard, North Holland, NL	3.00 GiB	0.00%	3.00 GiB	0.00%
f024184	Seoul, Seoul, KR	2.00 GiB	0.00%	2.00 GiB	0.00%
f020378	Unknown	2.00 GiB	0.00%	2.00 GiB	0.00%
f01345523	Antwerpen, Flanders, BE	2.00 GiB	0.00%	2.00 GiB	0.00%
f01784458	Oslo, Oslo, NO	1.00 GiB	0.00%	1.00 GiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

Since this is the 4th allocation, the following restrictions have been relaxed:

No more than 50% of unique data are stored with less than 4 providers.

⚠️ 83.14% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
26.40 TiB	26.41 TiB	1	11.45%
28.93 TiB	58.10 TiB	2	25.20%
35.01 TiB	107.20 TiB	3	46.49%
8.64 TiB	38.87 TiB	4	16.86%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Verifier
f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy	`Glif auto verified`	208.00 GiB	1	Jonathan Schwartz
f3wkp4blevjsrtbc6vwgjf2sedzjwsqmj3wsh4uex bp4k7dggs72kbvuv7xivsnz7cnmfazpmqp3qmchmz ms6a	Unknown	208.00 GiB	1	Unknown
f3u5dehxxe2uvehitioxhwjp27wpv72hsnuqhtz6s ce2wzqv2skhguivnsvwbkwgczcc5x4qf6eeao34te jqdq	`Glif auto verified`	16.00 GiB	1	Jonathan Schwartz

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f01858410

Client address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

DataCap allocation requested

400TiB

Id

d0b0ceab-504d-4178-85da-1ac1f49c0291

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Last two approvers

Alex11801 & kernelogic

Rule to calculate the allocation request amount

400% of weekly dc amount requested

DataCap allocation requested

400TiB

Total DataCap granted for client so far

128GiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

4.99PiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
605	9	200TiB	18.82	46.26TiB

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: The Internet Archive
Client: f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Since this is the 4th allocation, the following restrictions have been relaxed:

Storage provider should not exceed 50% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ f020378 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01611097	San Clemente, California, US	73.78 TiB	29.58%	71.66 TiB	2.87%
f01910202	Philadelphia, Pennsylvania, US	49.03 TiB	19.66%	48.37 TiB	1.35%
f01873432	Las Vegas, Nevada, US	31.73 TiB	12.72%	30.14 TiB	5.00%
f01904630	Las Vegas, Nevada, US	26.65 TiB	10.68%	23.78 TiB	10.77%
f01882184	Singapore, Singapore, SG	18.81 TiB	7.54%	18.81 TiB	0.00%
f01826669	Philadelphia, Pennsylvania, US	18.29 TiB	7.33%	18.29 TiB	0.00%
f01851683`new`	Las Vegas, Nevada, US	11.52 TiB	4.62%	11.52 TiB	0.00%
f01883179`new`	Philadelphia, Pennsylvania, US	7.69 TiB	3.08%	7.69 TiB	0.00%
f01858429	Boston, Massachusetts, US	5.46 TiB	2.19%	5.44 TiB	0.29%
f01606675	Montréal, Quebec, CA	3.30 TiB	1.32%	3.30 TiB	0.00%
f01851060	Las Vegas, Nevada, US	2.05 TiB	0.82%	2.05 TiB	0.00%
f066596	San Diego, California, US	274.00 GiB	0.11%	274.00 GiB	0.00%
f01091840	Montréal, Quebec, CA	176.00 GiB	0.07%	160.00 GiB	9.09%
f01199442	Heerhugowaard, North Holland, NL	146.00 GiB	0.06%	130.00 GiB	10.96%
f02576	Copenhagen, Capital Region, DK	96.00 GiB	0.04%	96.00 GiB	0.00%
f0157535	Montréal, Quebec, CA	80.00 GiB	0.03%	80.00 GiB	0.00%
f0104671	Kawasaki, Kanagawa, JP	64.00 GiB	0.03%	64.00 GiB	0.00%
f019104	Montréal, Quebec, CA	53.00 GiB	0.02%	53.00 GiB	0.00%
f09848	Rancho Santa Margarita, California, US	48.00 GiB	0.02%	48.00 GiB	0.00%
f0165400	Montréal, Quebec, CA	48.00 GiB	0.02%	48.00 GiB	0.00%
f01207045	Heerhugowaard, North Holland, NL	32.00 GiB	0.01%	32.00 GiB	0.00%
f058369	Boston, Massachusetts, US	32.00 GiB	0.01%	32.00 GiB	0.00%
f010088	Everett, Washington, US	18.00 GiB	0.01%	18.00 GiB	0.00%
f0694396	Birmingham, England, GB	17.00 GiB	0.01%	17.00 GiB	0.00%
f019551	Birmingham, England, GB	16.00 GiB	0.01%	16.00 GiB	0.00%
f030379	Seoul, Seoul, KR	16.00 GiB	0.01%	16.00 GiB	0.00%
f010446	Zaventem, Flanders, BE	16.00 GiB	0.01%	16.00 GiB	0.00%
f01199430	Heerhugowaard, North Holland, NL	3.00 GiB	0.00%	3.00 GiB	0.00%
f024184	Seoul, Seoul, KR	2.00 GiB	0.00%	2.00 GiB	0.00%
f020378	Unknown	2.00 GiB	0.00%	2.00 GiB	0.00%
f01345523	Antwerpen, Flanders, BE	2.00 GiB	0.00%	2.00 GiB	0.00%
f01784458	Oslo, Oslo, NO	1.00 GiB	0.00%	1.00 GiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

Since this is the 4th allocation, the following restrictions have been relaxed:

No more than 50% of unique data are stored with less than 4 providers.

⚠️ 83.88% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
27.39 TiB	27.40 TiB	1	10.99%
31.92 TiB	64.15 TiB	2	25.72%
38.37 TiB	117.65 TiB	3	47.17%
8.95 TiB	40.21 TiB	4	16.12%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Verifier
f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy	`Glif auto verified`	208.00 GiB	1	Jonathan Schwartz
f3wkp4blevjsrtbc6vwgjf2sedzjwsqmj3wsh4uex bp4k7dggs72kbvuv7xivsnz7cnmfazpmqp3qmchmz ms6a	Unknown	208.00 GiB	1	Unknown
f3u5dehxxe2uvehitioxhwjp27wpv72hsnuqhtz6s ce2wzqv2skhguivnsvwbkwgczcc5x4qf6eeao34te jqdq	`Glif auto verified`	16.00 GiB	1	Jonathan Schwartz

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

parkan commented 1 year ago

hi @filplus-checker, thank you for that thorough analysis! we've just run out of our allotment for this tranche so it seems like a good opportunity to reflect on these numbers

I'd like to ask a few questions/share some details about our process:

we currently use a a 3x baseline replication factor for any given CID, in part because having 3+ healthy SPs in our active set on an ongoing basis has been difficult to maintain due to maintenance and other issues; please let me know if we should prioritize this -- my understanding has been that overall data replication should target 4+ SPs, not every CID but I am happy to adjust (my apologies if I have misread those terms)
any deal data shared with other clients beyond a small number of test CIDs at the early stages (when we may have used test wallets) is due to other parties replicating us, all data is fully original, for which I can provide ample forensics if needed
any duplicate deals are unfortunately due to tooling issues; we are looking into deploying singularity/evergreen/spade as better deal tracking but our infrastructure is fairly unique and makes using these tools difficult, we are currently making a best effort at avoiding (we don't want duplicate transfers either!)
availability of reliable and responsive SPs with good connectivity to California has continued to be an issue
outside of the above concerns, I believe we are fulfilling the spirit and intent of the datacap program, with 11 SPs at > 1TiB stored

given the above, could you please advise us how to proceed? the following remediatons stand out to me:

increase replication factor for all CIDs to 4x+; our internal analysis has indicated 3x as the desired target (given we have 2 internal copies), but we can adjust this for the purposes of the program, also see following
add more high quality SPs to our active set; we currently have 7 in round-robin (randomized) rotation but several of them frequently drop out due to various maintenance issues, we would gladly welcome more SPs especially those well-peered to HE/SFMIX/CENIC
create additional replicas with our offline SPs; we're currently in the process of transferring nearly the entire dataset to one such SP and are ready to seal in coming days

please let me know how we should proceed; we've put a lot of work into making our ingestion systems robust over the past year and are eager to continue

parkan commented 1 year ago

I should also add that we're assisting in onboarding data from FF based grants in the coming months

guidance on appropriate replication of that data to fulfill the criteria desired by the notaries is very much appreciated (multi PB scale potentially as well)

kernelogic commented 1 year ago

Thanks for the comprehensive explanation. Willing to continue support.

cryptowhizzard commented 1 year ago

Same here

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaced5koplnxkt5rxfemma363nz3d3raudqqa32tkakihmeyf7ncemms

Address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Datacap Allocated

400.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

d0b0ceab-504d-4178-85da-1ac1f49c0291

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaced5koplnxkt5rxfemma363nz3d3raudqqa32tkakihmeyf7ncemms

parkan commented 1 year ago

@kernelogic @cryptowhizzard I appreciate the support! looking forward to proceeding + hitting the targets :)

cryptowhizzard commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedkt4vv7oexwshlj5xlvqywzmc2prmw4ta34vxqebusvpl6p3jgdy

Address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Datacap Allocated

400.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

d0b0ceab-504d-4178-85da-1ac1f49c0291

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedkt4vv7oexwshlj5xlvqywzmc2prmw4ta34vxqebusvpl6p3jgdy

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f01858410

Client address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

DataCap allocation requested

800TiB

Id

74e1ec38-3b34-4956-9799-a7c6e03fad65

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1wp6zoxj7sydnrywvzp276x3gayghi7r6le4tcwy

Last two approvers

cryptowhizzard & kernelogic

Rule to calculate the allocation request amount

800% of weekly dc amount requested

DataCap allocation requested

800TiB

Total DataCap granted for client so far

128GiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

4.99PiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
3702	9	400TiB	21.41	764MiB

filecoin-project / filecoin-plus-large-datasets

[DataCap Application] Internet Archive #52

Large Dataset Notary Application

Core Information

Project details

Use-case details

DataCap allocation plan

Multisig Notary requested

Total DataCap requested

Expected weekly DataCap usage rate

DataCap Allocation requested

Multisig Notary address

Client address

DataCap allocation requested

Request Approved

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedfdxlezwbyddmhojnyctgby4w7hoegkegeuhmwrbswf6apli36hw

Request Approved

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea4i245gwngvcvgk2uiwyjhjgjjdsmtq7bxqjvl4a24ombs7q7fyy

DataCap Allocation requested

Multisig Notary address

Client address

DataCap allocation requested

Request Proposed

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceatbxp3jmxfffzhr55mwgggonthpssma2wztblf6zjdcupz27cig6

Request Approved

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedh6tobo6y67vo6yvd3tachiqr5yh2ydglwxategtinjxvkptn3xu

DataCap Allocation requested

Request number 3

Multisig Notary address

Client address

DataCap allocation requested

Stats & Info for DataCap Allocation

Multisig Notary address

Client address

Last two approvers

Rule to calculate the allocation request amount

DataCap allocation requested

Total DataCap granted for client so far

Datacap to be granted to reach the total amount requested by the client (5PiB)

Stats

Request Proposed

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecm3ozfwyt2cgjbk7x4rdgpbjn6k5hkkhpksj3iz46zeo6rlc4qig

Request Approved

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea43yzzirtqm6fd6y5uuoaukkz63pobeoyylwwuu6ybyat3xka4kk

DataCap and CID Checker Report[^1]

Storage Provider Distribution

Deal Data Replication

Deal Data Shared with other Clients

DataCap and CID Checker Report[^1]

Storage Provider Distribution

Deal Data Replication

Deal Data Shared with other Clients

DataCap and CID Checker Report[^1]

Storage Provider Distribution

Deal Data Replication

Deal Data Shared with other Clients