cryptowhizzard commented 2 years ago

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

Organization Name: Speedium network
Website / Social Media: www.speedium.nl
Total amount of DataCap being requested (between 500 TiB and 5 PiB): 5 PiB
Weekly allocation of DataCap requested (usually between 1-100TiB): 1 PiB
On-chain address for first allocation: f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Short addendum on this project

This application is a followup of our first. NexRad is 2 PiB +. We are building the dataset at the moment. This datacap is solely for replica distribution. Link -> https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/340

Share a brief history of your project and organization.

Speedium / Dcent has engaged in Slingshot starting 2.6. We have successfully stored more than 15 differerent datasets with 20+ different miners.

What is the primary source of funding for this project?

Company account

What other projects/ecosystem stakeholders is this project associated with?

Slingshot competition hosted by Protocol Labs

Use-case details

Describe the data being stored onto Filecoin

The [Next Generation Weather Radar](https://www.ncdc.noaa.gov/data-access/radar-data/nexrad) (NEXRAD) is a network of 160 high-resolution Doppler radar sites that detects precipitation and atmospheric movement and disseminates data in approximately 5 minute intervals from each site.

Where was the data in this dataset sourced from?

AWS

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this.

https://www.ncei.noaa.gov/products/radar-data/nexrad

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

Yes.

What is the expected retrieval frequency for this data?

Multiple times p/y

For how long do you plan to keep this dataset stored on Filecoin?

18 months or longer

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

EU / US / Australia

How will you be distributing your data to storage providers? Is there an offline data transfer process?

The data will be transferred both offline and online.

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

We have a few providers who have been working with us during Slingshot Restore program and we'd like to continue working with them for ongoing Slingshot competition.

How will you be distributing deals across storage providers?

Max 2 copy's per storage provider if stored on different miners / locations.

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

Yes, we have the resources.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

Sunnyiscoming commented 2 years ago

@cryptowhizzard Hi,

https://www.ncei.noaa.gov/products/radar-data/nexrad, Page or Resource Not Found (404 Error)
Please ensure that the total amount of DataCap being requested is 30 PiB. As you said, this datacap is solely for replica distribution, NexRad is 2 PiB +. It means you need more than 10 copies.
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/340 The datacap being allocated has not be used(https://filplus.d.interplanetary.one/clients?filter=f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy). Maybe you should apply when you have used up the datacap you got in this issue.

cryptowhizzard commented 2 years ago

Hi @Sunnyiscoming

Fair points.

1 -> https://registry.opendata.aws/noaa-nexrad/

2 -> I am not sure if it is 2 PB exactly , this dataset is growing over the course of months. It can be that i won't apply for the last tranche.

3 -> This is rather difficult. I can't start sealing #340 solely with one miner by our own as this would violate the program rules. This is the reason i am doing this applications in parallel.

Sunnyiscoming commented 2 years ago

I don't quite understand your third point. Why can't datacap of #340 be allocated to other storage providers？

cryptowhizzard commented 2 years ago

@Sunnyiscoming this can be done. But if i assign and they don't start in the same pace or rythm it will result in 1 provider taking the whole first allocation.

Sunnyiscoming commented 2 years ago

I will talk about this with governance team because I think it is a problem which can be solved.

raghavrmadya commented 2 years ago

Hi @Sunnyiscoming

Fair points.

1 -> https://registry.opendata.aws/noaa-nexrad/

2 -> I am not sure if it is 2 PB exactly , this dataset is growing over the course of months. It can be that i won't apply for the last tranche.

3 -> This is rather difficult. I can't start sealing #340 solely with one miner by our own as this would violate the program rules. This is the reason i am doing this applications in parallel.

Hi @cryptowhizzard , can you elaborate on the difficulties? I'm unsure why you would only be able to seal with one SP?

cryptowhizzard commented 2 years ago

@raghavrmadya because the dataset is the same size as the datacap. Only one can seal the first replica to get it on chain. This is no problem as i will do the replica’s in a separate LDN, i just want to give notification here.

Sunnyiscoming commented 2 years ago

If this problem can be solved. Maybe you can close this issue and continue #340

raghavrmadya commented 2 years ago

Hi @cryptowhizzard, we have a PR open that was presented at the last gov call to increase the request limit to 25 PiBs. Do you mind waiting until end of this week when we merge that or are there SPs waiting on you?

cryptowhizzard commented 2 years ago

Hi @raghavrmadya that is ok. I will also update the wallet address so it is a unique wallet address per datacap request.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

raghavrmadya commented 2 years ago

@cryptowhizzard, the increase for the limit is still under discussion. If you want to proceed, you can create multiple issues for this app to proceed

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

cryptowhizzard commented 2 years ago

Hello,

I updated the request.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

raghavrmadya commented 2 years ago

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

1PiB

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

256TiB

kernelogic commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedi6fsiv7ycthoio4ls5hybsr5ynsdeamsn2dyn3ql54rhilgs7yq

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

256.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedi6fsiv7ycthoio4ls5hybsr5ynsdeamsn2dyn3ql54rhilgs7yq

s0nik42 commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceas7ti2iwzqqavdittbejtxdud6ar5f4krqqxubo5nccz7wbyvby6

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

256.00TiB

Signer Address

f1wxhnytjmklj2czezaqcfl7eb4nkgmaxysnegwii

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceas7ti2iwzqqavdittbejtxdud6ar5f4krqqxubo5nccz7wbyvby6

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

512TiB

large-datacap-requests[bot] commented 2 years ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Last two approvers

s0nik42 & kernelogic

Rule to calculate the allocation request amount

10% of total dc amount requested

DataCap allocation requested

512TiB

Total DataCap granted for client so far

3.24PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

1.75PiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
109914	35	256TiB	8.83	127.99GiB

Reiers commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceccbwxqbtrn4bfaaviht5bjxvkdkce2dnhy77mtfqwb4ato6uwjpu

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

512.00TiB

Signer Address

f1oz43ckvmtxmmsfzqm6bpnemqlavz4ifyl524chq

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceccbwxqbtrn4bfaaviht5bjxvkdkce2dnhy77mtfqwb4ato6uwjpu

kernelogic commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacec3p3irclnfdgvpxq6pzgpplwspsa7s6ueienbstnq2cl5vf2foty

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

512.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec3p3irclnfdgvpxq6pzgpplwspsa7s6ueienbstnq2cl5vf2foty

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

1PiB

Id

8336a73b-1c0b-4fda-86c3-bba095825bd1

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Last two approvers

kernelogic & Reiers

Rule to calculate the allocation request amount

20% of total dc amount requested

DataCap allocation requested

1PiB

Total DataCap granted for client so far

3.74PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

1.25PiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
120620	35	512TiB	10.45	124.87TiB

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceaasd2apuxiyudvk43vlq6frrzquetz6574adlg2w7uw3itqcyd42

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

1.00PiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

8336a73b-1c0b-4fda-86c3-bba095825bd1

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceaasd2apuxiyudvk43vlq6frrzquetz6574adlg2w7uw3itqcyd42

mjroddy commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecozpjxr5tgxermgyw7mzc27xnkwupdx3naruiqvpnhuzxek7tzym

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

1.00PiB

Signer Address

f1ystxl2ootvpirpa7ebgwl7vlhwkbx2r4zjxwe5i

Id

8336a73b-1c0b-4fda-86c3-bba095825bd1

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecozpjxr5tgxermgyw7mzc27xnkwupdx3naruiqvpnhuzxek7tzym

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

2PiB

Id

aaa8902b-82da-475b-b369-baaec731da70

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Last two approvers

megtei & kernelogic

Rule to calculate the allocation request amount

40% of total dc amount requested

DataCap allocation requested

2PiB

Total DataCap granted for client so far

4.74PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

256.00TiB

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
138265	40	1PiB	11.78	237.46TiB

xinaxu commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebkztcn6mhjbhzxrovop7nj5w4jmu7a2gbvzjhhrubbc7mnsrf6tg

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

2.00PiB

Signer Address

f1k3ysofkrrmqcot6fkx4wnezpczlltpirmrpsgui

Id

aaa8902b-82da-475b-b369-baaec731da70

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebkztcn6mhjbhzxrovop7nj5w4jmu7a2gbvzjhhrubbc7mnsrf6tg

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedwwfd25ffvfs4xgsfpg3nyp7khmo4jfd437llkbxsiaskb5vkvwq

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

2.00PiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

aaa8902b-82da-475b-b369-baaec731da70

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedwwfd25ffvfs4xgsfpg3nyp7khmo4jfd437llkbxsiaskb5vkvwq

kernelogic commented 1 year ago

oops, i guess this is double proposed now.

flyworker commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecfcryrb6ocnopjx4kacts44gz5ebhkayewoeal46vgzcfuewcm76

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

2.00PiB

Signer Address

f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy

Id

aaa8902b-82da-475b-b369-baaec731da70

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecfcryrb6ocnopjx4kacts44gz5ebhkayewoeal46vgzcfuewcm76

filplus-checker commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: Speedium network
Client: f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy
Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Storage provider should not exceed 25% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ 25.80% of total deal sealed by f01157271 are duplicate data.

⚠️ 27.24% of total deal sealed by f01208803 are duplicate data.

⚠️ 24.50% of total deal sealed by f01208154 are duplicate data.

⚠️ 21.24% of total deal sealed by f01156835 are duplicate data.

⚠️ 52.64% of total deal sealed by f01208189 are duplicate data.

⚠️ 34.04% of total deal sealed by f01208042 are duplicate data.

⚠️ 23.73% of total deal sealed by f01156538 are duplicate data.

⚠️ f01908299 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01919423	Sydney, New South Wales, AU	597.30 TiB	11.11%	576.67 TiB	3.45%
f01908671	Philadelphia, Pennsylvania, US	503.09 TiB	9.36%	460.88 TiB	8.39%
f01938357	Sydney, New South Wales, AU	431.94 TiB	8.04%	431.91 TiB	0.01%
f01851482	Busan, Busan, KR	383.06 TiB	7.13%	383.06 TiB	0.00%
f01852664	Singapore, Singapore, SG	306.91 TiB	5.71%	306.91 TiB	0.00%
f01882184	Singapore, Singapore, SG	296.03 TiB	5.51%	296.03 TiB	0.00%
f01880047	Singapore, Singapore, SG	259.88 TiB	4.84%	259.88 TiB	0.00%
f01877571	Singapore, Singapore, SG	253.53 TiB	4.72%	253.38 TiB	0.06%
f01967501`new`	Philadelphia, Pennsylvania, US	196.28 TiB	3.65%	196.28 TiB	0.00%
f01720359	Heerhugowaard, North Holland, NL	184.78 TiB	3.44%	179.19 TiB	3.03%
f01938665	Sham Shui Po, Sham Shui Po, HK	172.31 TiB	3.21%	170.25 TiB	1.20%
f01938717	Singapore, Singapore, SG	164.16 TiB	3.05%	163.94 TiB	0.13%
f01878005	Singapore, Singapore, SG	146.69 TiB	2.73%	146.19 TiB	0.34%
f01882177	Singapore, Singapore, SG	142.47 TiB	2.65%	142.41 TiB	0.04%
f01786387	Heerhugowaard, North Holland, NL	114.28 TiB	2.13%	110.69 TiB	3.14%
f01157271	Sydney, New South Wales, AU	99.56 TiB	1.85%	73.88 TiB	25.80%
f01208803	Sydney, New South Wales, AU	98.19 TiB	1.83%	71.44 TiB	27.24%
f01771403	Heerhugowaard, North Holland, NL	96.50 TiB	1.80%	89.53 TiB	7.22%
f01208154	Sydney, New South Wales, AU	85.19 TiB	1.58%	64.31 TiB	24.50%
f01059489	Plano, Texas, US	83.31 TiB	1.55%	83.31 TiB	0.00%
f01981059	Richardson, Texas, US	75.78 TiB	1.41%	75.78 TiB	0.00%
f01208632	Sydney, New South Wales, AU	68.91 TiB	1.28%	55.13 TiB	20.00%
f01156835`new`	Sydney, New South Wales, AU	64.75 TiB	1.20%	51.00 TiB	21.24%
f01208189	Sydney, New South Wales, AU	64.00 TiB	1.19%	30.31 TiB	52.64%
f01207874	Sydney, New South Wales, AU	62.69 TiB	1.17%	53.91 TiB	14.01%
f01208042	Sydney, New South Wales, AU	56.38 TiB	1.05%	37.19 TiB	34.04%
f01157249	Sydney, New South Wales, AU	51.91 TiB	0.97%	46.56 TiB	10.30%
f01207954	Sydney, New South Wales, AU	45.97 TiB	0.86%	42.44 TiB	7.68%
f01156883	Sydney, New South Wales, AU	28.13 TiB	0.52%	26.84 TiB	4.56%
f01199430	Heerhugowaard, North Holland, NL	27.09 TiB	0.50%	24.44 TiB	9.80%
f01937642`new`	Heerhugowaard, North Holland, NL	26.66 TiB	0.50%	26.00 TiB	2.46%
f01156538`new`	Sydney, New South Wales, AU	23.44 TiB	0.44%	17.88 TiB	23.73%
f01206408	Sydney, New South Wales, AU	22.13 TiB	0.41%	20.41 TiB	7.77%
f01157018	Sydney, New South Wales, AU	20.91 TiB	0.39%	20.91 TiB	0.00%
f01908299	Unknown	20.44 TiB	0.38%	20.38 TiB	0.31%
f01154023	Sydney, New South Wales, AU	19.84 TiB	0.37%	18.63 TiB	6.14%
f01156975	Sydney, New South Wales, AU	19.22 TiB	0.36%	19.19 TiB	0.16%
f01157027	Sydney, New South Wales, AU	18.34 TiB	0.34%	18.13 TiB	1.19%
f022352	Oslo, Oslo, NO	17.25 TiB	0.32%	17.06 TiB	1.09%
f0855584`new`	Lincoln, Nebraska, US	11.28 TiB	0.21%	11.28 TiB	0.00%
f01156901	Sydney, New South Wales, AU	10.72 TiB	0.20%	10.69 TiB	0.29%
f01864434	Sydney, New South Wales, AU	3.22 TiB	0.06%	3.22 TiB	0.00%
f01878201	Hangzhou, Zhejiang, CN	288.00 GiB	0.01%	288.00 GiB	0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

No more than 25% of unique data are stored with less than 4 providers.

✔️ Data replication looks healthy.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
24.11 TiB	24.52 TiB	1	0.46%
180.94 TiB	371.00 TiB	2	6.90%
245.81 TiB	784.75 TiB	3	14.60%
209.22 TiB	879.75 TiB	4	16.37%
176.78 TiB	937.72 TiB	5	17.45%
121.94 TiB	766.06 TiB	6	14.25%
103.53 TiB	757.56 TiB	7	14.09%
63.19 TiB	533.59 TiB	8	9.93%
24.88 TiB	237.09 TiB	9	4.41%
7.09 TiB	75.69 TiB	10	1.41%
640.00 GiB	7.03 TiB	11	0.13%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Verifier
f1ws3n5tuxtyg26lraqkjirz7qon7y7ckju7hhmii	Tech Greedy Inc	4.48 PiB	13,100	LDN v3 multisig
f1pc5usvsbfgxxbq7c7quhhg6k7l6y5reiwqr3noy	Speedium network	245.34 TiB	4,493	LDN v3 multisig
f1z7jogzx4x42wtilzb4lu6iotlad5rptt2acbzpi	Speedium network	5.91 TiB	9	LDN v3 multisig
f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq	Speedium NetworkSpeedium Network	832.00 GiB	9	LDN # 77

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

1.25PiB

Id

9310b40a-68b7-45f9-b58a-786862ad7fec

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Last two approvers

flyworker & kernelogic

Rule to calculate the allocation request amount

80% of total dc amount requested

DataCap allocation requested

1.25PiB

Total DataCap granted for client so far

6.74PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

-1970324836974591B

Stats

Number of deals	Number of storage providers	Previous DC Allocated	Top provider	Remaining DC
211756	49	2PiB	9.27	480.32TiB

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Organization: Speedium network
Client: f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy
Approvers

1flyworker
4kernelogic
1mjroddy
1Reiers
1s0nik42
1xinaxu

Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

Storage provider should not exceed 30% of total datacap.
Storage provider should not be storing duplicate data for more than 20%.
Storage provider should have published its public IP address.
All storage providers should be located in different regions.

⚠️ 27.57% of total deal sealed by f01208803 are duplicate data.

⚠️ 28.09% of total deal sealed by f01157271 are duplicate data.

⚠️ 24.28% of total deal sealed by f01208154 are duplicate data.

⚠️ 27.18% of total deal sealed by f01208632 are duplicate data.

⚠️ 50.24% of total deal sealed by f01208189 are duplicate data.

⚠️ 24.42% of total deal sealed by f01156835 are duplicate data.

⚠️ 34.04% of total deal sealed by f01208042 are duplicate data.

⚠️ f01908299 has unknown IP location.

Provider	Location	Total Deals Sealed	Percentage	Unique Data	Duplicate Deals
f01919423	Sydney, New South Wales, AU `Andrew Sjoquist Enterprises Pty Ltd`	597.30 TiB	9.05%	576.67 TiB	3.45%
f01938357	Sydney, New South Wales, AU `Andrew Sjoquist Enterprises Pty Ltd`	431.94 TiB	6.55%	431.91 TiB	0.01%
f01864434	Sydney, New South Wales, AU `Andrew Sjoquist Enterprises Pty Ltd`	3.22 TiB	0.05%	3.22 TiB	0.00%
f01208803	Sydney, New South Wales, AU `Anycast Global Backbone`	106.91 TiB	1.62%	77.44 TiB	27.57%
f01157271	Sydney, New South Wales, AU `Anycast Global Backbone`	103.81 TiB	1.57%	74.66 TiB	28.09%
f01208154	Sydney, New South Wales, AU `Anycast Global Backbone`	86.09 TiB	1.31%	65.19 TiB	24.28%
f01208632	Sydney, New South Wales, AU `Anycast Global Backbone`	77.84 TiB	1.18%	56.69 TiB	27.18%
f01208189	Sydney, New South Wales, AU `Anycast Global Backbone`	77.38 TiB	1.17%	38.50 TiB	50.24%
f01156835`new`	Sydney, New South Wales, AU `Anycast Global Backbone`	69.22 TiB	1.05%	52.31 TiB	24.42%
f01157249	Sydney, New South Wales, AU `Anycast Global Backbone`	55.63 TiB	0.84%	49.72 TiB	10.62%
f01156883	Sydney, New South Wales, AU `Anycast Global Backbone`	40.31 TiB	0.61%	38.22 TiB	5.19%
f01156538`new`	Sydney, New South Wales, AU `Anycast Global Backbone`	33.66 TiB	0.51%	27.47 TiB	18.38%
f01157027	Sydney, New South Wales, AU `Anycast Global Backbone`	31.34 TiB	0.48%	25.28 TiB	19.34%
f01156975	Sydney, New South Wales, AU `Anycast Global Backbone`	30.88 TiB	0.47%	25.78 TiB	16.50%
f01156901	Sydney, New South Wales, AU `Anycast Global Backbone`	28.13 TiB	0.43%	26.53 TiB	5.67%
f01157018	Sydney, New South Wales, AU `Anycast Global Backbone`	26.75 TiB	0.41%	23.34 TiB	12.73%
f01154023	Sydney, New South Wales, AU `Anycast Global Backbone`	19.84 TiB	0.30%	18.63 TiB	6.14%
f01981059	Irving, Texas, US `AT&T Services, Inc.`	202.56 TiB	3.07%	202.56 TiB	0.00%
f022352	Oslo, Oslo, NO `Blix Solutions AS`	17.25 TiB	0.26%	17.06 TiB	1.09%
f01938665	Sham Shui Po, Sham Shui Po, HK `China Unicom Global`	172.31 TiB	2.61%	170.25 TiB	1.20%
f01059489	Plano, Texas, US `Cogent Communications`	147.78 TiB	2.24%	147.78 TiB	0.00%
f01996927`new`	Plano, Texas, US `Cogent Communications`	49.06 TiB	0.74%	49.06 TiB	0.00%
f01906874	Plano, Texas, US `Cogent Communications`	2.50 TiB	0.04%	2.50 TiB	0.00%
f01207874	Sydney, New South Wales, AU `DATACOM SYSTEMS (AU) PTY LTD`	62.69 TiB	0.95%	53.91 TiB	14.01%
f01208042	Sydney, New South Wales, AU `DATACOM SYSTEMS (AU) PTY LTD`	56.38 TiB	0.85%	37.19 TiB	34.04%
f01207954	Sydney, New South Wales, AU `DATACOM SYSTEMS (AU) PTY LTD`	45.97 TiB	0.70%	42.44 TiB	7.68%
f01206408	Sydney, New South Wales, AU `DATACOM SYSTEMS (AU) PTY LTD`	22.13 TiB	0.34%	20.41 TiB	7.77%
f01992630	Louisville, Kentucky, US `Flexential Colorado Corp.`	123.41 TiB	1.87%	123.41 TiB	0.00%
f01971600	Louisville, Kentucky, US `Flexential Colorado Corp.`	104.75 TiB	1.59%	104.75 TiB	0.00%
f01878201	Hangzhou, Zhejiang, CN `Hangzhou Alibaba Advertising Co.,Ltd.`	288.00 GiB	0.00%	288.00 GiB	0.00%
f01518369	San Jose, California, US `HONG KONG Megalayer Technology Co.,Limited`	297.22 TiB	4.51%	297.22 TiB	0.00%
f01851482	Busan, Busan, KR `Korea Telecom`	383.06 TiB	5.81%	383.06 TiB	0.00%
f0855584`new`	Lincoln, Nebraska, US `LightEdge Solutions`	50.03 TiB	0.76%	50.03 TiB	0.00%
f01091851	Lincoln, Nebraska, US `LightEdge Solutions`	18.28 TiB	0.28%	18.28 TiB	0.00%
f01736668	Lincoln, Nebraska, US `LightEdge Solutions`	17.63 TiB	0.27%	17.63 TiB	0.00%
f01882184	Herndon, Virginia, US `PCCW Global, Inc.`	296.03 TiB	4.49%	296.03 TiB	0.00%
f01880047	Herndon, Virginia, US `PCCW Global, Inc.`	259.88 TiB	3.94%	259.88 TiB	0.00%
f01877571	Herndon, Virginia, US `PCCW Global, Inc.`	253.53 TiB	3.84%	253.38 TiB	0.06%
f01878005	Herndon, Virginia, US `PCCW Global, Inc.`	146.69 TiB	2.22%	146.19 TiB	0.34%
f01882177	Herndon, Virginia, US `PCCW Global, Inc.`	142.47 TiB	2.16%	142.41 TiB	0.04%
f01852664	Singapore, Singapore, SG `StarHub Ltd`	306.91 TiB	4.65%	306.91 TiB	0.00%
f01938717	Singapore, Singapore, SG `StarHub Ltd`	164.16 TiB	2.49%	163.94 TiB	0.13%
f01908671	New York City, New York, US `Unitas Global LLC`	503.09 TiB	7.63%	460.88 TiB	8.39%
f01967501`new`	New York City, New York, US `Unitas Global LLC`	461.16 TiB	6.99%	461.16 TiB	0.00%
f01720359	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	184.78 TiB	2.80%	179.19 TiB	3.03%
f01786387	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	114.28 TiB	1.73%	110.69 TiB	3.14%
f01771403	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	96.50 TiB	1.46%	89.53 TiB	7.22%
f01199430	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	27.09 TiB	0.41%	24.44 TiB	9.80%
f01937642`new`	Heerhugowaard, North Holland, NL `Wijnand Schouten trading as Speedium`	26.66 TiB	0.40%	26.00 TiB	2.46%
f01908299	Unknown `Unknown`	20.44 TiB	0.31%	20.38 TiB	0.31%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

No more than 30% of unique data are stored with less than 4 providers.

✔️ Data replication looks healthy.

Unique Data Size	Total Deals Made	Number of Providers	Deal Percentage
12.39 TiB	12.45 TiB	1	0.19%
69.50 TiB	143.66 TiB	2	2.18%
167.03 TiB	528.06 TiB	3	8.00%
203.91 TiB	861.66 TiB	4	13.06%
184.19 TiB	972.59 TiB	5	14.74%
167.63 TiB	1.02 PiB	6	15.88%
128.94 TiB	943.94 TiB	7	14.31%
106.72 TiB	892.56 TiB	8	13.53%
65.69 TiB	621.84 TiB	9	9.43%
34.66 TiB	362.09 TiB	10	5.49%
14.25 TiB	163.91 TiB	11	2.48%
3.22 TiB	40.28 TiB	12	0.61%
480.00 GiB	6.25 TiB	13	0.09%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

However, this could be possible if all below clients use same software to prepare for the exact same dataset or they belong to a series of LDN applications for the same dataset.

⚠️ CID sharing has been observed.

Other Client	Application	Total Deals Affected	Unique CIDs	Approvers
f1ws3n5tuxtyg26lraqkjirz7qon7y7ckju7hhmii	Tech Greedy Inc	4.65 PiB	13,100	`5`cryptowhizzard `4`Fenbushi-Filecoin `1`IreneYoung `1`jamerduhgamer `6`kernelogic `1`MegTei `1`s0nik42 `3`xingjitansuo
f1pc5usvsbfgxxbq7c7quhhg6k7l6y5reiwqr3noy	Speedium network	245.34 TiB	4,493	`3`kernelogic `1`liyunzhi-666 `1`MegTei `1`neogeweb3 `2`Reiers `3`s0nik42 `1`xingjitansuo
f1z7jogzx4x42wtilzb4lu6iotlad5rptt2acbzpi	Speedium network	6.19 TiB	9	`1`flyworker `1`kernelogic `4`MegTei `2`psh0691 `3`Reiers `3`s0nik42
f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq	Speedium NetworkSpeedium Network	832.00 GiB	9	`2`dannyob `3`flyworker `4`MegTei `3`Reiers `6`s0nik42

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

Kevin-FF-USA commented 1 year ago

Datacap Request Trigger

Total DataCap requested

1 PiB

Expected weekly DataCap usage rate

400 TiB

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

256TiB

Id

feee3cd5-ac51-47fa-9810-cf45b5f1223a

NiwanDao commented 1 year ago

@cryptowhizzard The distribution looks healthy, but can you please explain the CID sharing with other entity https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/335 ?

cryptowhizzard commented 1 year ago

@xingjitansuo

Yes i can. #335 and we did build exactly the same dataset from AWS S3. This means the same CID's will be created.

NiwanDao commented 1 year ago

The same CID only happens when the Data preparer uses the same way on the same raw data to prepare the car file.

cryptowhizzard commented 1 year ago

@xingjitansuo yes , we did. We build it with Singularity, so did Xinan.

NiwanDao commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceapnxsq4z54ed4fxvihz4bjuy2dhycfz4sdkqtolw4exp5x3hcyeu

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

256.00TiB

Signer Address

f1a2lia2cwwekeubwo4nppt4v4vebxs2frozarz3q

Id

feee3cd5-ac51-47fa-9810-cf45b5f1223a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceapnxsq4z54ed4fxvihz4bjuy2dhycfz4sdkqtolw4exp5x3hcyeu

xinaxu commented 1 year ago

checker:manualTrigger f1ws3n5tuxtyg26lraqkjirz7qon7y7ckju7hhmii f1pc5usvsbfgxxbq7c7quhhg6k7l6y5reiwqr3noy

filecoin-project / filecoin-plus-large-datasets

[DataCap Application] Speedium - NexRad V2 [ Replica distribution ] Part 1 / 5 #483

Large Dataset Notary Application

Core Information

Project details

Use-case details

DataCap allocation plan

Datacap Request Trigger

DataCap Allocation requested

Multisig Notary address

Client address

DataCap allocation requested

Request Proposed

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedi6fsiv7ycthoio4ls5hybsr5ynsdeamsn2dyn3ql54rhilgs7yq

Request Approved

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceas7ti2iwzqqavdittbejtxdud6ar5f4krqqxubo5nccz7wbyvby6

DataCap Allocation requested

Request number 2

Multisig Notary address

Client address

DataCap allocation requested

Stats & Info for DataCap Allocation

Multisig Notary address

Client address

Last two approvers

Rule to calculate the allocation request amount

DataCap allocation requested

Total DataCap granted for client so far

Datacap to be granted to reach the total amount requested by the client (5 PiB)

Stats

Request Proposed

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceccbwxqbtrn4bfaaviht5bjxvkdkce2dnhy77mtfqwb4ato6uwjpu

Request Approved

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec3p3irclnfdgvpxq6pzgpplwspsa7s6ueienbstnq2cl5vf2foty

DataCap Allocation requested

Request number 3

Multisig Notary address

Client address

DataCap allocation requested

Id

Stats & Info for DataCap Allocation

Multisig Notary address

Client address

Last two approvers

Rule to calculate the allocation request amount

DataCap allocation requested

Total DataCap granted for client so far

Datacap to be granted to reach the total amount requested by the client (5 PiB)

Stats

Request Proposed

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

Id

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceaasd2apuxiyudvk43vlq6frrzquetz6574adlg2w7uw3itqcyd42

Request Approved

Message sent to Filecoin Network

Address

Datacap Allocated

Signer Address

Id

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecozpjxr5tgxermgyw7mzc27xnkwupdx3naruiqvpnhuzxek7tzym

DataCap Allocation requested