[DataCap Application] Speedium - NIH NCBI Sequence Read Archive [2 / 27]

cryptowhizzard commented 1 year ago

Data Owner Name

NIH - National Institute of Health

Data Owner Country/Region

United States

Data Owner Industry

Life Science / Healthcare

Website

https://www.nih.gov/

Social Media

https://www.facebook.com/nih.gov/

Total amount of DataCap being requested

5PiB

Weekly allocation of DataCap requested

500TiB

On-chain address for first allocation

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

Custom multisig

[ ] Use Custom Multisig

Identifier

No response

Share a brief history of your project and organization

Since its launch, the Filecoin network has become an important player in the decentralised storage space, offering a secure and transparent alternative to traditional data storage solutions.

We as Speedium / DCENT have been engaged with storing real and valuable datasets on the Filecoin network since Slingshot 2.6 and have been actively developing tools to improve the process. We are always on the lookout for new and useful client data to onboard.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

NIH NCBI Sequence Read Archive (SRA) on AWS
The Sequence Read Archive (SRA), produced by the [National Center for Biotechnology Information (NCBI)](https://www.ncbi.nlm.nih.gov/) at the [National Library of Medicine (NLM)](http://nlm.nih.gov/) at the [National Institutes of Health (NIH)](http://www.nih.gov/), stores raw DNA sequencing data and alignment information from high-throughput sequencing platforms.

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

No response

How do you plan to prepare the dataset

IPFS, lotus, singularity, graphsplit

If you answered "other/custom tool" in the previous question, enter the details here

No response

Please share a sample of the data

https://registry.opendata.aws/ncbi-sra/

Confirm that this is a public dataset that can be retrieved by anyone on the Network

[X] I confirm

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

1.5 to 2 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, North America, South America, Europe, Australia (continent)

How will you be distributing your data to storage providers

HTTP or FTP server, IPFS, Shipping hard drives, Lotus built-in data transfer

How do you plan to choose storage providers

Slack, Big data exchange, Partners

If you answered "Others" in the previous question, what is the tool or platform you plan to use

No response

If you already have a list of storage providers to work with, fill out their names and provider IDs below

MinerID	City	Continent	Business/Entity
`f01944347`	Oregon	USA	Jenny, Dabai
`f01952350`	Oregon	USA	Jenny, Dabai
`f01972364`	Oregon	USA	Jenny, Dabai
`f01972376`	Oregon	USA	Jenny, Dabai
`f02000937`	Chengdu	CN	MTY
`f01915033`	Chengdu	CN	MTY
`f0120****`	Melbourne	AU	HOLON
`f0115****`	Melbourne	AU	HOLON
`f01199430`	Heerhugowaard	EU	DCENT
`f01786387`	Heerhugowaard	EU	DCENT
`f01201327`	Heerhugowaard	EU	DCENT
`f01937642`	Heerhugowaard	EU	DCENT
`f0198****`	Dallas	USA	GREATERHEAT
`f0188****`	Singapore	AS	GREATERHEAT
`f01091851`	Omaha	USA	DLTx
`f01736668`	Omaha	USA	DLTx
`f01820744`	Omaha	USA	DLTx
`f0855584`	Omaha	USA	DLTx
`f01794610`	Omaha	USA	DLTx
`f01838599`	Kansas City	USA	DLTx
`f01845552`	Kansas City	USA	DLTx

How do you plan to make deals to your storage providers

Boost client, Lotus client, Singularity

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

cryptowhizzard commented 1 year ago

(Proposal https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1511 is broken this is a re-apply of that request)

simonkim0515 commented 1 year ago

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

500TiB

Client address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

DataCap allocation requested

250TiB

Id

5fa80a2b-ff53-4fc3-92b4-42512ce1de2c

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

There is no previous allocation for this issue.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

flyworker commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacea3ly6ai5oahpddveomkc3niczqnmkhlqqgvuu2oapze3uiqlybfm

Address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

Datacap Allocated

250.00TiB

Signer Address

f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy

Id

5fa80a2b-ff53-4fc3-92b4-42512ce1de2c

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea3ly6ai5oahpddveomkc3niczqnmkhlqqgvuu2oapze3uiqlybfm

herrehesse commented 1 year ago

Kindly requesting the assistance of a Notary in signing the next tranche of the datacap. @kernelogic @s0nik42 @steven004 @GaryGJG @mjroddy @xinaxu @xingjitansuo

kernelogic commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebmr4oiyxpefsgz5dd3eqc7bcwfin3nvrudrh4lvibl2z5vjfhzck

Address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

Datacap Allocated

250.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebmr4oiyxpefsgz5dd3eqc7bcwfin3nvrudrh4lvibl2z5vjfhzck

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

DataCap allocation requested

500TiB

Id

64eb75dd-84a2-485f-b200-ae702cd3352e

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

Last two approvers

kernelogic & flyworker

Rule to calculate the allocation request amount

100% of weekly dc amount requested

DataCap allocation requested

500TiB

Total DataCap granted for client so far

4.26PiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

749.28TiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
139269	24	250TiB	22.42	56.72TiB

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: NIH- National Institute of Health
Client: f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy
Approvers

1flyworker
1kernelogic

Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Since this is the 3rd allocation, the following restrictions have been relaxed:

Storage provider should not exceed 70% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ 28.19% of total deal sealed by f01208803 are duplicate data.

⚠️ 41.21% of total deal sealed by f01208189 are duplicate data.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01156975	Melbourne, Victoria, AU `Anycast Global Backbone`	59.94 TiB	1.46%	49.66 TiB	17.15%
f01157271	Melbourne, Victoria, AU `Anycast Global Backbone`	56.15 TiB	1.37%	54.37 TiB	3.17%
f01208632	Melbourne, Victoria, AU `Anycast Global Backbone`	51.92 TiB	1.27%	50.61 TiB	2.53%
f01156901	Melbourne, Victoria, AU `Anycast Global Backbone`	48.39 TiB	1.18%	41.61 TiB	14.01%
f01208803	Melbourne, Victoria, AU `Anycast Global Backbone`	45.01 TiB	1.10%	32.32 TiB	28.19%
f01208189	Melbourne, Victoria, AU `Anycast Global Backbone`	44.33 TiB	1.08%	26.06 TiB	41.21%
f01157018	Melbourne, Victoria, AU `Anycast Global Backbone`	44.28 TiB	1.08%	42.72 TiB	3.53%
f01157027	Melbourne, Victoria, AU `Anycast Global Backbone`	39.09 TiB	0.95%	37.37 TiB	4.40%
f01157249	Melbourne, Victoria, AU `Anycast Global Backbone`	37.39 TiB	0.91%	36.52 TiB	2.34%
f01156835	Melbourne, Victoria, AU `Anycast Global Backbone`	17.88 TiB	0.44%	17.26 TiB	3.49%
f01208154	Melbourne, Victoria, AU `Anycast Global Backbone`	16.26 TiB	0.40%	16.20 TiB	0.38%
f01156538	Melbourne, Victoria, AU `Anycast Global Backbone`	12.41 TiB	0.30%	12.41 TiB	0.00%
f022352	Oslo, Oslo, NO `Blix Solutions AS`	75.38 TiB	1.84%	68.56 TiB	9.04%
f02000937	Chengdu, Sichuan, CN `China Mobile Communications Group Co., Ltd.`	152.33 TiB	3.72%	152.33 TiB	0.00%
f01972376`new`	Maywood Park, Oregon, US `Flexential Colorado Corp.`	962.98 TiB	23.49%	962.32 TiB	0.07%
f01972364`new`	Maywood Park, Oregon, US `Flexential Colorado Corp.`	926.84 TiB	22.60%	926.84 TiB	0.00%
f01952350	Maywood Park, Oregon, US `Flexential Colorado Corp.`	238.13 TiB	5.81%	236.38 TiB	0.73%
f01944347	Maywood Park, Oregon, US `Flexential Colorado Corp.`	226.63 TiB	5.53%	226.63 TiB	0.00%
f01392893	Amsterdam, North Holland, NL `Fusix Networks B.V.`	11.49 TiB	0.28%	11.49 TiB	0.00%
f01199430	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	582.29 TiB	14.20%	575.54 TiB	1.16%
f01786387	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	197.76 TiB	4.82%	193.13 TiB	2.34%
f01201327	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	134.84 TiB	3.29%	134.84 TiB	0.00%
f01937642	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	64.63 TiB	1.58%	61.50 TiB	4.84%
f01771403	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	54.05 TiB	1.32%	54.05 TiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

Since this is the 3rd allocation, the following restrictions have been relaxed:

No more than 70% of unique data are stored with less than 3 providers.

✔️ Data replication looks healthy.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
1.53 PiB	1.53 PiB	1	38.23%
477.73 TiB	959.78 TiB	2	23.41%
258.87 TiB	800.80 TiB	3	19.53%
68.36 TiB	288.33 TiB	4	7.03%
29.81 TiB	158.16 TiB	5	3.86%
29.66 TiB	195.03 TiB	6	4.76%
17.41 TiB	129.78 TiB	7	3.17%
128.00 GiB	1.06 TiB	8	0.03%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

However, this could be possible if all below clients use same software to prepare for the exact same dataset or they belong to a series of LDN applications for the same dataset.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Approvers
f1z7jogzx4x42wtilzb4lu6iotlad5rptt2acbzpi	Speedium network	44.17 TiB	1,341	`1`flyworker `1`kernelogic `4`MegTei `2`psh0691 `3`Reiers `3`s0nik42

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

psh0691 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedfbq22nyqpysjcvdpoycrgn44qotamexyqz3faq2ij5uz7ctfcaa

Address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

Datacap Allocated

250.00TiB

Signer Address

f1qdko4jg25vo35qmyvcrw4ak4fmuu3f5rif2kc7i

Id

64eb75dd-84a2-485f-b200-ae702cd3352e

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedfbq22nyqpysjcvdpoycrgn44qotamexyqz3faq2ij5uz7ctfcaa

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

DataCap allocation requested

1000.0TiB

Id

bc899abf-3c08-458e-8a33-ce3e720e98aa

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

Last two approvers

psh0691 & kernelogic

Rule to calculate the allocation request amount

200% of weekly dc amount requested

DataCap allocation requested

1000.0TiB

Total DataCap granted for client so far

4.26PiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

749.28TiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
139269	24	500TiB	22.42	46.81TiB

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: NIH- National Institute of Health
Client: f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy
Approvers

1flyworker
1kernelogic
1psh0691

Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Storage provider should not exceed 30% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ 28.19% of total deal sealed by f01208803 are duplicate data.

⚠️ 41.21% of total deal sealed by f01208189 are duplicate data.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01156975	Melbourne, Victoria, AU `Anycast Global Backbone`	59.94 TiB	1.46%	49.66 TiB	17.15%
f01157271	Melbourne, Victoria, AU `Anycast Global Backbone`	56.15 TiB	1.37%	54.37 TiB	3.17%
f01208632	Melbourne, Victoria, AU `Anycast Global Backbone`	51.92 TiB	1.26%	50.61 TiB	2.53%
f01156901	Melbourne, Victoria, AU `Anycast Global Backbone`	48.39 TiB	1.18%	41.61 TiB	14.01%
f01208803	Melbourne, Victoria, AU `Anycast Global Backbone`	45.01 TiB	1.10%	32.32 TiB	28.19%
f01208189	Melbourne, Victoria, AU `Anycast Global Backbone`	44.33 TiB	1.08%	26.06 TiB	41.21%
f01157018	Melbourne, Victoria, AU `Anycast Global Backbone`	44.28 TiB	1.08%	42.72 TiB	3.53%
f01157027	Melbourne, Victoria, AU `Anycast Global Backbone`	39.09 TiB	0.95%	37.37 TiB	4.40%
f01157249	Melbourne, Victoria, AU `Anycast Global Backbone`	37.92 TiB	0.92%	37.05 TiB	2.31%
f01156835	Melbourne, Victoria, AU `Anycast Global Backbone`	17.88 TiB	0.44%	17.26 TiB	3.49%
f01208154	Melbourne, Victoria, AU `Anycast Global Backbone`	16.26 TiB	0.40%	16.20 TiB	0.38%
f01156538	Melbourne, Victoria, AU `Anycast Global Backbone`	12.51 TiB	0.30%	12.51 TiB	0.00%
f022352	Oslo, Oslo, NO `Blix Solutions AS`	75.78 TiB	1.85%	68.97 TiB	8.99%
f02000937	Chengdu, Sichuan, CN `China Mobile Communications Group Co., Ltd.`	152.36 TiB	3.71%	152.36 TiB	0.00%
f01915033	Chengdu, Sichuan, CN `China Mobile Communications Group Co., Ltd.`	640.00 GiB	0.02%	640.00 GiB	0.00%
f01972376`new`	Maywood Park, Oregon, US `Flexential Colorado Corp.`	962.98 TiB	23.45%	962.32 TiB	0.07%
f01972364`new`	Maywood Park, Oregon, US `Flexential Colorado Corp.`	926.84 TiB	22.57%	926.84 TiB	0.00%
f01952350	Maywood Park, Oregon, US `Flexential Colorado Corp.`	238.13 TiB	5.80%	236.38 TiB	0.73%
f01944347	Maywood Park, Oregon, US `Flexential Colorado Corp.`	226.63 TiB	5.52%	226.63 TiB	0.00%
f01392893	Amsterdam, North Holland, NL `Fusix Networks B.V.`	11.49 TiB	0.28%	11.49 TiB	0.00%
f01199430	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	582.29 TiB	14.18%	575.54 TiB	1.16%
f01786387	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	197.76 TiB	4.81%	193.13 TiB	2.34%
f01201327	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	134.84 TiB	3.28%	134.84 TiB	0.00%
f01937642	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	64.63 TiB	1.57%	61.50 TiB	4.84%
f01771403	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	59.11 TiB	1.44%	59.11 TiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

No more than 30% of unique data are stored with less than 4 providers.

⚠️ 80.77% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
1.53 PiB	1.53 PiB	1	38.16%
476.02 TiB	956.34 TiB	2	23.28%
256.55 TiB	793.67 TiB	3	19.32%
72.36 TiB	304.48 TiB	4	7.41%
29.72 TiB	157.69 TiB	5	3.84%
29.91 TiB	196.56 TiB	6	4.79%
17.44 TiB	130.00 TiB	7	3.17%
128.00 GiB	1.06 TiB	8	0.03%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

However, this could be possible if all below clients use same software to prepare for the exact same dataset or they belong to a series of LDN applications for the same dataset.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Approvers
f1z7jogzx4x42wtilzb4lu6iotlad5rptt2acbzpi	Speedium network	44.17 TiB	1,341	`1`flyworker `1`kernelogic `4`MegTei `2`psh0691 `3`Reiers `3`s0nik42

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

NDLABS-Leo commented 1 year ago

Hidde Hoogland contacted me on slack and ND reviewed #1554 in accordance with LDN's notary review criteria and communicated the results to Speedium via slack. speedium promptly troubleshot the search issue and provided a searchable proof of the figure. As the check bot indicates, the LDN performed well and ND will continue to follow up and we are willing to support if the retrieval is good.

cryptowhizzard commented 1 year ago

@NDLABS-OFFICE Transparency is appreciated!

large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

jhookersyd commented 1 year ago

Hi All / @NDLABS-OFFICE / @cryptowhizzard / @herrehesse,

We are currently moving from the MKT Node to Boost over the next 8 days, we have 47 miners so it will take us a little time to make sure we're back up and running 100%.

With this downtime, we'll also be upgrading switches and NICs across our whole architecture. During this maintenance period, we will stop ingesting deals on miners as the update process takes place and retrievals will be intermittent. We'll endeavour to be back online on the 13th of Feb.

We can't wait to get on BOOST!!!!

Thank you! Jonathan

large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

cryptowhizzard commented 1 year ago

Kindly requesting the assistance of a Notary in signing the next tranche of the datacap. @kernelogic @s0nik42 @steven004 @GaryGJG @mjroddy @xinaxu @xingjitansuo

xinaxu commented 1 year ago

For all different copies in AU, are they storing the same file or are you splitting the dataset to store with different miners? I think having 12 copies within a city does not make much sense here.

herrehesse commented 1 year ago

@xinaxu they are splitting the same dataset across different miners. 12 copies in the same city does not make sense. For more information you can ask @jhookersyd.

jhookersyd commented 1 year ago

Morning All

Correct. We take one dataset and split it across many miners in the same Datacentre this allows us to update miners while still sealing on other miners. Happy to give people a DC tour if anyone is in Sydney.

Thanks! Jonathan

xinaxu commented 1 year ago

So I have already signed. but it does not appear here for some reason. https://filfox.info/en/message/bafy2bzacede4trrquhvplv57jrd2vabvn27zf7myqovj2uywiowyfciq7cu7u

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f02049625

Client address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

DataCap allocation requested

1.95PiB

Id

6b1e3e35-3e21-4160-b8e7-b312c960931a

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

Last two approvers

psh0691 & kernelogic

Rule to calculate the allocation request amount

400% of weekly dc amount requested

DataCap allocation requested

1.95PiB

Total DataCap granted for client so far

4.51PiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

499.28TiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
139269	24	1000.0TiB	22.42	238.98TiB

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

No client address found for this issue.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

NDLABS-Leo commented 1 year ago

@cryptowhizzard thanks. @jhookersyd Thank you for your work.

s0nik42 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebu2xpjtwmj5f3nh5qdzei6cgn4yh7ybxjuzjeskyeasjmvbib5na

Address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

Datacap Allocated

1.95PiB

Signer Address

f1wxhnytjmklj2czezaqcfl7eb4nkgmaxysnegwii

Id

6b1e3e35-3e21-4160-b8e7-b312c960931a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebu2xpjtwmj5f3nh5qdzei6cgn4yh7ybxjuzjeskyeasjmvbib5na

large-datacap-requests[bot] commented 1 year ago

There was an error signing the transaction. The message cid: bafy2bzacebu2xpjtwmj5f3nh5qdzei6cgn4yh7ybxjuzjeskyeasjmvbib5na

Please, contact the governance team.

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

No client address found for this issue.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

fabriziogianni7 commented 1 year ago

unblocking it manually till the error is fixed

herrehesse commented 1 year ago

@fabriziogianni7 Manually unblocked.

Kindly requesting the assistance of a Notary in signing the new tranche of the datacap. @kernelogic @s0nik42 @steven004 @GaryGJG @mjroddy @xinaxu @xingjitansuo

xinaxu commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedm4qeaa7vl2og6ytgucdnunzmow2sv3hrt7chl77g2ya3cqdkw3i

Address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

Datacap Allocated

1.95PiB

Signer Address

f1k3ysofkrrmqcot6fkx4wnezpczlltpirmrpsgui

Id

6b1e3e35-3e21-4160-b8e7-b312c960931a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedm4qeaa7vl2og6ytgucdnunzmow2sv3hrt7chl77g2ya3cqdkw3i

xinaxu commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: NIH- National Institute of Health
Client: f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy
Approvers

1flyworker
1kernelogic
1psh0691
1s0nik42
1xinaxu

Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Storage provider should not exceed 30% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ 24.49% of total deal sealed by f01208803 are duplicate data.

⚠️ 35.66% of total deal sealed by f01208189 are duplicate data.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01156975	Melbourne, Victoria, AU `Anycast Global Backbone`	59.94 TiB	1.35%	49.66 TiB	17.15%
f01208632	Melbourne, Victoria, AU `Anycast Global Backbone`	59.07 TiB	1.33%	57.75 TiB	2.22%
f01157271	Melbourne, Victoria, AU `Anycast Global Backbone`	56.15 TiB	1.27%	54.37 TiB	3.17%
f01208803	Melbourne, Victoria, AU `Anycast Global Backbone`	51.80 TiB	1.17%	39.11 TiB	24.49%
f01208189	Melbourne, Victoria, AU `Anycast Global Backbone`	51.22 TiB	1.16%	32.95 TiB	35.66%
f01156901	Melbourne, Victoria, AU `Anycast Global Backbone`	48.42 TiB	1.09%	41.64 TiB	14.00%
f01157018	Melbourne, Victoria, AU `Anycast Global Backbone`	44.28 TiB	1.00%	42.72 TiB	3.53%
f01157249	Melbourne, Victoria, AU `Anycast Global Backbone`	42.92 TiB	0.97%	42.05 TiB	2.04%
f01157027	Melbourne, Victoria, AU `Anycast Global Backbone`	39.09 TiB	0.88%	37.37 TiB	4.40%
f01156835	Melbourne, Victoria, AU `Anycast Global Backbone`	17.88 TiB	0.40%	17.26 TiB	3.49%
f01208154	Melbourne, Victoria, AU `Anycast Global Backbone`	17.10 TiB	0.39%	17.04 TiB	0.37%
f01156538	Melbourne, Victoria, AU `Anycast Global Backbone`	15.20 TiB	0.34%	15.20 TiB	0.00%
f022352	Oslo, Oslo, NO `Blix Solutions AS`	77.31 TiB	1.74%	70.50 TiB	8.81%
f02000937	Chengdu, Sichuan, CN `China Mobile Communications Group Co., Ltd.`	290.40 TiB	6.55%	290.40 TiB	0.00%
f01915033	Chengdu, Sichuan, CN `China Mobile Communications Group Co., Ltd.`	94.50 TiB	2.13%	94.50 TiB	0.00%
f01972376`new`	Maywood Park, Oregon, US `Flexential Colorado Corp.`	962.98 TiB	21.72%	962.32 TiB	0.07%
f01972364`new`	Maywood Park, Oregon, US `Flexential Colorado Corp.`	926.84 TiB	20.91%	926.84 TiB	0.00%
f01952350	Maywood Park, Oregon, US `Flexential Colorado Corp.`	238.13 TiB	5.37%	236.38 TiB	0.73%
f01944347	Maywood Park, Oregon, US `Flexential Colorado Corp.`	226.63 TiB	5.11%	226.63 TiB	0.00%
f01392893	Amsterdam, North Holland, NL `Fusix Networks B.V.`	56.44 TiB	1.27%	56.44 TiB	0.00%
f01199430	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	582.29 TiB	13.14%	575.54 TiB	1.16%
f01786387	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	197.76 TiB	4.46%	193.13 TiB	2.34%
f01201327	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	134.84 TiB	3.04%	134.84 TiB	0.00%
f01771403	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	77.05 TiB	1.74%	77.05 TiB	0.00%
f01937642	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	64.63 TiB	1.46%	61.50 TiB	4.84%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

No more than 30% of unique data are stored with less than 4 providers.

⚠️ 67.38% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
1.50 PiB	1.50 PiB	1	34.59%
423.97 TiB	848.48 TiB	2	19.14%
198.78 TiB	605.43 TiB	3	13.66%
198.13 TiB	822.61 TiB	4	18.56%
53.64 TiB	280.93 TiB	5	6.34%
31.00 TiB	202.91 TiB	6	4.58%
18.25 TiB	136.00 TiB	7	3.07%
416.00 GiB	3.34 TiB	8	0.08%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

However, this could be possible if all below clients use same software to prepare for the exact same dataset or they belong to a series of LDN applications for the same dataset.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Approvers
f1z7jogzx4x42wtilzb4lu6iotlad5rptt2acbzpi	Speedium network	44.17 TiB	1,341	`1`flyworker `1`kernelogic `4`MegTei `2`psh0691 `3`Reiers `3`s0nik42

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

dongpo313 commented 1 year ago

@xinaxu @s0nik42 It was observed that the cid was repeated, but you signed it. Do you think this is responsible?

cryptowhizzard commented 1 year ago

Schermafbeelding 2023-02-07 om 15 48 38

filplusapp commented 1 year ago

ERROR: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid baga6ea4seaqfr3xvzgwf3a3unn6qlqjczonbencfmgdn6y2t42r4zaw7c3bocja: getting pieces containing block baga6ea4seaqfr3xvzgwf3a3unn6qlqjczonbencfmgdn6y2t42r4zaw7c3bocja: failed to lookup index for mh 92202058eef5c9ac5d83746b7d05c122cb9a1234456186df6353e6a3cc82df16c2e124, err: datastore: key not found

ERROR: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid baga6ea4seaqh7wu5s5rsqcv6jy7uq5lejcl2m7fz3rat4mzdixiqauvc5c4o4ey: getting pieces containing block baga6ea4seaqh7wu5s5rsqcv6jy7uq5lejcl2m7fz3rat4mzdixiqauvc5c4o4ey: failed to lookup index for mh 9220207fda9d9763280abe4e3f4875644897a67cb9dc413e332345d10052a2e8b8ee13, err: datastore: key not found

ERROR: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid baga6ea4seaqpnx2mpwls7uwxfznpzaiizsaqxaynd3gyre56mah4ivgt4gcrqkq: getting pieces containing block baga6ea4seaqpnx2mpwls7uwxfznpzaiizsaqxaynd3gyre56mah4ivgt4gcrqkq: failed to lookup index for mh 922020f6df4c7d972fd2d72e5afc8108cc810b830d1ecd8893be600fc454d3e185182a, err: datastore: key not found

cryptowhizzard commented 1 year ago

ERROR: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid baga6ea4seaqfr3xvzgwf3a3unn6qlqjczonbencfmgdn6y2t42r4zaw7c3bocja: getting pieces containing block baga6ea4seaqfr3xvzgwf3a3unn6qlqjczonbencfmgdn6y2t42r4zaw7c3bocja: failed to lookup index for mh 92202058eef5c9ac5d83746b7d05c122cb9a1234456186df6353e6a3cc82df16c2e124, err: datastore: key not found

ERROR: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid baga6ea4seaqh7wu5s5rsqcv6jy7uq5lejcl2m7fz3rat4mzdixiqauvc5c4o4ey: getting pieces containing block baga6ea4seaqh7wu5s5rsqcv6jy7uq5lejcl2m7fz3rat4mzdixiqauvc5c4o4ey: failed to lookup index for mh 9220207fda9d9763280abe4e3f4875644897a67cb9dc413e332345d10052a2e8b8ee13, err: datastore: key not found

ERROR: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid baga6ea4seaqpnx2mpwls7uwxfznpzaiizsaqxaynd3gyre56mah4ivgt4gcrqkq: getting pieces containing block baga6ea4seaqpnx2mpwls7uwxfznpzaiizsaqxaynd3gyre56mah4ivgt4gcrqkq: failed to lookup index for mh 922020f6df4c7d972fd2d72e5afc8108cc810b830d1ecd8893be600fc454d3e185182a, err: datastore: key not found

These deals are just being sealed on chain. When the sealing is done the deals will be avaiable.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f02049625

Client address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

DataCap allocation requested

1.34PiB

Id

d858ee79-a0dc-491a-8fc5-d644496bc544

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy

Last two approvers

xinaxu & s0nik42

Rule to calculate the allocation request amount

800% of weekly dc amount requested

DataCap allocation requested

1.34PiB

Total DataCap granted for client so far

6.46PiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

-1646540652827115B

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
186327	33	1.95PiB	16.76	474.74TiB

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: NIH- National Institute of Health
Client: f1mgnwoczfj25foxn4555wvwyak6rsynzy7z73azy
Approvers

1flyworker
1kernelogic
1psh0691
1s0nik42
1xinaxu

Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Storage provider should not exceed 30% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ 28.90% of total deal sealed by f01208189 are duplicate data.

⚠️ 23.04% of total deal sealed by f01208803 are duplicate data.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01157271	Melbourne, Victoria, AU `Anycast Global Backbone`	71.45 TiB	1.26%	69.67 TiB	2.49%
f01208632	Melbourne, Victoria, AU `Anycast Global Backbone`	66.24 TiB	1.17%	64.93 TiB	1.98%
f01208189	Melbourne, Victoria, AU `Anycast Global Backbone`	63.21 TiB	1.11%	44.95 TiB	28.90%
f01156975	Melbourne, Victoria, AU `Anycast Global Backbone`	59.94 TiB	1.06%	49.66 TiB	17.15%
f01156901	Melbourne, Victoria, AU `Anycast Global Backbone`	58.89 TiB	1.04%	52.11 TiB	11.51%
f01208803	Melbourne, Victoria, AU `Anycast Global Backbone`	55.07 TiB	0.97%	42.38 TiB	23.04%
f01157018	Melbourne, Victoria, AU `Anycast Global Backbone`	52.77 TiB	0.93%	51.20 TiB	2.96%
f01157027	Melbourne, Victoria, AU `Anycast Global Backbone`	46.46 TiB	0.82%	44.74 TiB	3.70%
f01157249	Melbourne, Victoria, AU `Anycast Global Backbone`	42.92 TiB	0.76%	42.05 TiB	2.04%
f01208154	Melbourne, Victoria, AU `Anycast Global Backbone`	18.65 TiB	0.33%	18.59 TiB	0.34%
f01156835	Melbourne, Victoria, AU `Anycast Global Backbone`	17.88 TiB	0.32%	17.26 TiB	3.49%
f01156538	Melbourne, Victoria, AU `Anycast Global Backbone`	15.26 TiB	0.27%	15.26 TiB	0.00%
f022352	Oslo, Oslo, NO `Blix Solutions AS`	77.31 TiB	1.36%	70.50 TiB	8.81%
f02000937	Chengdu, Sichuan, CN `China Mobile Communications Group Co., Ltd.`	446.48 TiB	7.88%	446.48 TiB	0.00%
f01915033	Chengdu, Sichuan, CN `China Mobile Communications Group Co., Ltd.`	94.50 TiB	1.67%	94.50 TiB	0.00%
f02026193`new`	Chengdu, Sichuan, CN `China Mobile Communications Group Co., Ltd.`	53.49 TiB	0.94%	53.49 TiB	0.00%
f01345523	Antwerpen, Flanders, BE `Cogent Communications`	9.71 TiB	0.17%	9.71 TiB	0.00%
f01972376`new`	Maywood Park, Oregon, US `Flexential Colorado Corp.`	962.98 TiB	16.99%	962.32 TiB	0.07%
f01972364`new`	Maywood Park, Oregon, US `Flexential Colorado Corp.`	926.84 TiB	16.35%	926.84 TiB	0.00%
f01992630	Dallas, Texas, US `Flexential Colorado Corp.`	279.64 TiB	4.93%	279.64 TiB	0.00%
f01971600	Dallas, Texas, US `Flexential Colorado Corp.`	262.19 TiB	4.62%	262.19 TiB	0.00%
f01952350	Maywood Park, Oregon, US `Flexential Colorado Corp.`	238.13 TiB	4.20%	236.38 TiB	0.73%
f01944347	Maywood Park, Oregon, US `Flexential Colorado Corp.`	226.63 TiB	4.00%	226.63 TiB	0.00%
f02031042`new`	Maywood Park, Oregon, US `Flexential Colorado Corp.`	98.30 TiB	1.73%	98.30 TiB	0.00%
f01392893	Amsterdam, North Holland, NL `Fusix Networks B.V.`	268.28 TiB	4.73%	268.28 TiB	0.00%
f01907545	Hong Kong, Central and Western, HK `HK Broadband Network Ltd.`	67.72 TiB	1.19%	67.72 TiB	0.00%
f01889910	Phoenix, Arizona, US `Level 3 Parent, LLC`	24.78 TiB	0.44%	24.78 TiB	0.00%
f01847751	Denver, Colorado, US `Level 3 Parent, LLC`	7.16 TiB	0.13%	7.16 TiB	0.00%
f01199430	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	582.29 TiB	10.27%	575.54 TiB	1.16%
f01786387	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	197.76 TiB	3.49%	193.13 TiB	2.34%
f01201327	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	134.84 TiB	2.38%	134.84 TiB	0.00%
f01771403	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	77.05 TiB	1.36%	77.05 TiB	0.00%
f01937642	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	64.63 TiB	1.14%	61.50 TiB	4.84%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

No more than 30% of unique data are stored with less than 4 providers.

⚠️ 49.90% of deals are for data replicated across less than 4 storage providers.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
1.43 PiB	1.43 PiB	1	25.89%
312.90 TiB	626.30 TiB	2	11.05%
244.41 TiB	734.67 TiB	3	12.96%
235.39 TiB	950.96 TiB	4	16.77%
192.10 TiB	986.09 TiB	5	17.39%
78.21 TiB	481.77 TiB	6	8.50%
32.00 TiB	243.12 TiB	7	4.29%
16.44 TiB	139.84 TiB	8	2.47%
3.94 TiB	36.53 TiB	9	0.64%
224.00 GiB	2.25 TiB	10	0.04%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

However, this could be possible if all below clients use same software to prepare for the exact same dataset or they belong to a series of LDN applications for the same dataset.[^3]

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Approvers
f1z7jogzx4x42wtilzb4lu6iotlad5rptt2acbzpi	Speedium network	44.17 TiB	1,341	`1`flyworker `1`kernelogic `4`MegTei `2`psh0691 `3`Reiers `3`s0nik42

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

herrehesse commented 1 year ago

Kindly requesting the assistance of a Notary in signing the new tranche of the datacap. @kernelogic @s0nik42 @steven004 @GaryGJG @mjroddy @xinaxu @xingjitansuo

flyworker commented 1 year ago

No more than 30% of unique data are stored with less than 4 providers. ⚠️ 49.90% of deals are for data replicated across less than 4 storage providers. CID sharing has been observed.

Any explanation?

herrehesse commented 1 year ago

@flyworker Thanks for your due diligence.

The SP's "Jenny & Dabai" have been sealing and downloading the NiH set a lot quicker than the rest of the SP's we contacted. At present, GreaterHeat and Holon are working rapidly to reduce the distinctive data distribution to below 30%. I anticipate that the next report from the CID checker will demonstrate that this goal has been achieved.

flyworker commented 1 year ago

Do you have an estimated of time of completion?

filecoin-project / filecoin-plus-large-datasets