filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] Speedium - NexRad V2 [ Replica distribution ] Part 1 / 5 #483

Closed cryptowhizzard closed 1 year ago

cryptowhizzard commented 2 years ago

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Short addendum on this project

This application is a followup of our first. NexRad is 2 PiB +. We are building the dataset at the moment. This datacap is solely for replica distribution. Link -> https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/340

Share a brief history of your project and organization.

Speedium / Dcent has engaged in Slingshot starting 2.6. We have successfully stored more than 15 differerent datasets with 20+ different miners.

What is the primary source of funding for this project?

Company account

What other projects/ecosystem stakeholders is this project associated with?

Slingshot competition hosted by Protocol Labs

Use-case details

Describe the data being stored onto Filecoin

The [Next Generation Weather Radar](https://www.ncdc.noaa.gov/data-access/radar-data/nexrad) (NEXRAD) is a network of 160 high-resolution Doppler radar sites that detects precipitation and atmospheric movement and disseminates data in approximately 5 minute intervals from each site.

Where was the data in this dataset sourced from?

AWS

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this.

https://www.ncei.noaa.gov/products/radar-data/nexrad

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

Yes.

What is the expected retrieval frequency for this data?

Multiple times p/y

For how long do you plan to keep this dataset stored on Filecoin?

18 months or longer

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

EU / US / Australia

How will you be distributing your data to storage providers? Is there an offline data transfer process?

The data will be transferred both offline and online.

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

We have a few providers who have been working with us during Slingshot Restore program and we'd like to continue working with them for ongoing Slingshot competition.

How will you be distributing deals across storage providers?

Max 2 copy's per storage provider if stored on different miners / locations.

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

Yes, we have the resources.
large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

Sunnyiscoming commented 2 years ago

@cryptowhizzard Hi,

  1. https://www.ncei.noaa.gov/products/radar-data/nexrad, Page or Resource Not Found (404 Error)
  2. Please ensure that the total amount of DataCap being requested is 30 PiB. As you said, this datacap is solely for replica distribution, NexRad is 2 PiB +. It means you need more than 10 copies.
  3. https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/340 The datacap being allocated has not be used(https://filplus.d.interplanetary.one/clients?filter=f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy). Maybe you should apply when you have used up the datacap you got in this issue.
cryptowhizzard commented 2 years ago

Hi @Sunnyiscoming

Fair points.

1 -> https://registry.opendata.aws/noaa-nexrad/

2 -> I am not sure if it is 2 PB exactly , this dataset is growing over the course of months. It can be that i won't apply for the last tranche.

3 -> This is rather difficult. I can't start sealing #340 solely with one miner by our own as this would violate the program rules. This is the reason i am doing this applications in parallel.

Sunnyiscoming commented 2 years ago

I don't quite understand your third point. Why can't datacap of #340 be allocated to other storage providers?

cryptowhizzard commented 2 years ago

@Sunnyiscoming this can be done. But if i assign and they don't start in the same pace or rythm it will result in 1 provider taking the whole first allocation.

Sunnyiscoming commented 2 years ago

I will talk about this with governance team because I think it is a problem which can be solved.

raghavrmadya commented 2 years ago

Hi @Sunnyiscoming

Fair points.

1 -> https://registry.opendata.aws/noaa-nexrad/

2 -> I am not sure if it is 2 PB exactly , this dataset is growing over the course of months. It can be that i won't apply for the last tranche.

3 -> This is rather difficult. I can't start sealing #340 solely with one miner by our own as this would violate the program rules. This is the reason i am doing this applications in parallel.

Hi @cryptowhizzard , can you elaborate on the difficulties? I'm unsure why you would only be able to seal with one SP?

cryptowhizzard commented 2 years ago

@raghavrmadya because the dataset is the same size as the datacap. Only one can seal the first replica to get it on chain. This is no problem as i will do the replica’s in a separate LDN, i just want to give notification here.

Sunnyiscoming commented 2 years ago

If this problem can be solved. Maybe you can close this issue and continue #340

raghavrmadya commented 2 years ago

Hi @cryptowhizzard, we have a PR open that was presented at the last gov call to increase the request limit to 25 PiBs. Do you mind waiting until end of this week when we merge that or are there SPs waiting on you?

cryptowhizzard commented 2 years ago

Hi @raghavrmadya that is ok. I will also update the wallet address so it is a unique wallet address per datacap request.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

raghavrmadya commented 1 year ago

@cryptowhizzard, the increase for the limit is still under discussion. If you want to proceed, you can create multiple issues for this app to proceed

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

cryptowhizzard commented 1 year ago

Hello,

I updated the request.

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

raghavrmadya commented 1 year ago

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

1PiB

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

256TiB

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedi6fsiv7ycthoio4ls5hybsr5ynsdeamsn2dyn3ql54rhilgs7yq

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

256.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedi6fsiv7ycthoio4ls5hybsr5ynsdeamsn2dyn3ql54rhilgs7yq

s0nik42 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceas7ti2iwzqqavdittbejtxdud6ar5f4krqqxubo5nccz7wbyvby6

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

256.00TiB

Signer Address

f1wxhnytjmklj2czezaqcfl7eb4nkgmaxysnegwii

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceas7ti2iwzqqavdittbejtxdud6ar5f4krqqxubo5nccz7wbyvby6

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

512TiB

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Last two approvers

s0nik42 & kernelogic

Rule to calculate the allocation request amount

10% of total dc amount requested

DataCap allocation requested

512TiB

Total DataCap granted for client so far

3.24PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

1.75PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
109914 35 256TiB 8.83 127.99GiB
Reiers commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceccbwxqbtrn4bfaaviht5bjxvkdkce2dnhy77mtfqwb4ato6uwjpu

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

512.00TiB

Signer Address

f1oz43ckvmtxmmsfzqm6bpnemqlavz4ifyl524chq

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceccbwxqbtrn4bfaaviht5bjxvkdkce2dnhy77mtfqwb4ato6uwjpu

kernelogic commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacec3p3irclnfdgvpxq6pzgpplwspsa7s6ueienbstnq2cl5vf2foty

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

512.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec3p3irclnfdgvpxq6pzgpplwspsa7s6ueienbstnq2cl5vf2foty

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

1PiB

Id

8336a73b-1c0b-4fda-86c3-bba095825bd1

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Last two approvers

kernelogic & Reiers

Rule to calculate the allocation request amount

20% of total dc amount requested

DataCap allocation requested

1PiB

Total DataCap granted for client so far

3.74PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

1.25PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
120620 35 512TiB 10.45 124.87TiB
kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceaasd2apuxiyudvk43vlq6frrzquetz6574adlg2w7uw3itqcyd42

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

1.00PiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

8336a73b-1c0b-4fda-86c3-bba095825bd1

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceaasd2apuxiyudvk43vlq6frrzquetz6574adlg2w7uw3itqcyd42

mjroddy commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecozpjxr5tgxermgyw7mzc27xnkwupdx3naruiqvpnhuzxek7tzym

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

1.00PiB

Signer Address

f1ystxl2ootvpirpa7ebgwl7vlhwkbx2r4zjxwe5i

Id

8336a73b-1c0b-4fda-86c3-bba095825bd1

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecozpjxr5tgxermgyw7mzc27xnkwupdx3naruiqvpnhuzxek7tzym

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

2PiB

Id

aaa8902b-82da-475b-b369-baaec731da70

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Last two approvers

megtei & kernelogic

Rule to calculate the allocation request amount

40% of total dc amount requested

DataCap allocation requested

2PiB

Total DataCap granted for client so far

4.74PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

256.00TiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
138265 40 1PiB 11.78 237.46TiB
xinaxu commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebkztcn6mhjbhzxrovop7nj5w4jmu7a2gbvzjhhrubbc7mnsrf6tg

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

2.00PiB

Signer Address

f1k3ysofkrrmqcot6fkx4wnezpczlltpirmrpsgui

Id

aaa8902b-82da-475b-b369-baaec731da70

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebkztcn6mhjbhzxrovop7nj5w4jmu7a2gbvzjhhrubbc7mnsrf6tg

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedwwfd25ffvfs4xgsfpg3nyp7khmo4jfd437llkbxsiaskb5vkvwq

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

2.00PiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

aaa8902b-82da-475b-b369-baaec731da70

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedwwfd25ffvfs4xgsfpg3nyp7khmo4jfd437llkbxsiaskb5vkvwq

kernelogic commented 1 year ago

oops, i guess this is double proposed now.

flyworker commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecfcryrb6ocnopjx4kacts44gz5ebhkayewoeal46vgzcfuewcm76

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

2.00PiB

Signer Address

f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy

Id

aaa8902b-82da-475b-b369-baaec731da70

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecfcryrb6ocnopjx4kacts44gz5ebhkayewoeal46vgzcfuewcm76

filplus-checker commented 1 year ago

DataCap and CID Checker Report[^1]

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

⚠️ 25.80% of total deal sealed by f01157271 are duplicate data.

⚠️ 27.24% of total deal sealed by f01208803 are duplicate data.

⚠️ 24.50% of total deal sealed by f01208154 are duplicate data.

⚠️ 21.24% of total deal sealed by f01156835 are duplicate data.

⚠️ 52.64% of total deal sealed by f01208189 are duplicate data.

⚠️ 34.04% of total deal sealed by f01208042 are duplicate data.

⚠️ 23.73% of total deal sealed by f01156538 are duplicate data.

⚠️ f01908299 has unknown IP location.

Provider Location Total Deals Sealed Percentage Unique Data Duplicate Deals
f01919423 Sydney, New South Wales, AU 597.30 TiB 11.11% 576.67 TiB 3.45%
f01908671 Philadelphia, Pennsylvania, US 503.09 TiB 9.36% 460.88 TiB 8.39%
f01938357 Sydney, New South Wales, AU 431.94 TiB 8.04% 431.91 TiB 0.01%
f01851482 Busan, Busan, KR 383.06 TiB 7.13% 383.06 TiB 0.00%
f01852664 Singapore, Singapore, SG 306.91 TiB 5.71% 306.91 TiB 0.00%
f01882184 Singapore, Singapore, SG 296.03 TiB 5.51% 296.03 TiB 0.00%
f01880047 Singapore, Singapore, SG 259.88 TiB 4.84% 259.88 TiB 0.00%
f01877571 Singapore, Singapore, SG 253.53 TiB 4.72% 253.38 TiB 0.06%
f01967501new Philadelphia, Pennsylvania, US 196.28 TiB 3.65% 196.28 TiB 0.00%
f01720359 Heerhugowaard, North Holland, NL 184.78 TiB 3.44% 179.19 TiB 3.03%
f01938665 Sham Shui Po, Sham Shui Po, HK 172.31 TiB 3.21% 170.25 TiB 1.20%
f01938717 Singapore, Singapore, SG 164.16 TiB 3.05% 163.94 TiB 0.13%
f01878005 Singapore, Singapore, SG 146.69 TiB 2.73% 146.19 TiB 0.34%
f01882177 Singapore, Singapore, SG 142.47 TiB 2.65% 142.41 TiB 0.04%
f01786387 Heerhugowaard, North Holland, NL 114.28 TiB 2.13% 110.69 TiB 3.14%
f01157271 Sydney, New South Wales, AU 99.56 TiB 1.85% 73.88 TiB 25.80%
f01208803 Sydney, New South Wales, AU 98.19 TiB 1.83% 71.44 TiB 27.24%
f01771403 Heerhugowaard, North Holland, NL 96.50 TiB 1.80% 89.53 TiB 7.22%
f01208154 Sydney, New South Wales, AU 85.19 TiB 1.58% 64.31 TiB 24.50%
f01059489 Plano, Texas, US 83.31 TiB 1.55% 83.31 TiB 0.00%
f01981059 Richardson, Texas, US 75.78 TiB 1.41% 75.78 TiB 0.00%
f01208632 Sydney, New South Wales, AU 68.91 TiB 1.28% 55.13 TiB 20.00%
f01156835new Sydney, New South Wales, AU 64.75 TiB 1.20% 51.00 TiB 21.24%
f01208189 Sydney, New South Wales, AU 64.00 TiB 1.19% 30.31 TiB 52.64%
f01207874 Sydney, New South Wales, AU 62.69 TiB 1.17% 53.91 TiB 14.01%
f01208042 Sydney, New South Wales, AU 56.38 TiB 1.05% 37.19 TiB 34.04%
f01157249 Sydney, New South Wales, AU 51.91 TiB 0.97% 46.56 TiB 10.30%
f01207954 Sydney, New South Wales, AU 45.97 TiB 0.86% 42.44 TiB 7.68%
f01156883 Sydney, New South Wales, AU 28.13 TiB 0.52% 26.84 TiB 4.56%
f01199430 Heerhugowaard, North Holland, NL 27.09 TiB 0.50% 24.44 TiB 9.80%
f01937642new Heerhugowaard, North Holland, NL 26.66 TiB 0.50% 26.00 TiB 2.46%
f01156538new Sydney, New South Wales, AU 23.44 TiB 0.44% 17.88 TiB 23.73%
f01206408 Sydney, New South Wales, AU 22.13 TiB 0.41% 20.41 TiB 7.77%
f01157018 Sydney, New South Wales, AU 20.91 TiB 0.39% 20.91 TiB 0.00%
f01908299 Unknown 20.44 TiB 0.38% 20.38 TiB 0.31%
f01154023 Sydney, New South Wales, AU 19.84 TiB 0.37% 18.63 TiB 6.14%
f01156975 Sydney, New South Wales, AU 19.22 TiB 0.36% 19.19 TiB 0.16%
f01157027 Sydney, New South Wales, AU 18.34 TiB 0.34% 18.13 TiB 1.19%
f022352 Oslo, Oslo, NO 17.25 TiB 0.32% 17.06 TiB 1.09%
f0855584new Lincoln, Nebraska, US 11.28 TiB 0.21% 11.28 TiB 0.00%
f01156901 Sydney, New South Wales, AU 10.72 TiB 0.20% 10.69 TiB 0.29%
f01864434 Sydney, New South Wales, AU 3.22 TiB 0.06% 3.22 TiB 0.00%
f01878201 Hangzhou, Zhejiang, CN 288.00 GiB 0.01% 288.00 GiB 0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

✔️ Data replication looks healthy.

Unique Data Size Total Deals Made Number of Providers Deal Percentage
24.11 TiB 24.52 TiB 1 0.46%
180.94 TiB 371.00 TiB 2 6.90%
245.81 TiB 784.75 TiB 3 14.60%
209.22 TiB 879.75 TiB 4 16.37%
176.78 TiB 937.72 TiB 5 17.45%
121.94 TiB 766.06 TiB 6 14.25%
103.53 TiB 757.56 TiB 7 14.09%
63.19 TiB 533.59 TiB 8 9.93%
24.88 TiB 237.09 TiB 9 4.41%
7.09 TiB 75.69 TiB 10 1.41%
640.00 GiB 7.03 TiB 11 0.13%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client Application Total Deals Affected Unique CIDs Verifier
f1ws3n5tuxtyg26lraqkjirz7qon7y7ckju7hhmii Tech Greedy Inc 4.48 PiB 13,100 LDN v3 multisig
f1pc5usvsbfgxxbq7c7quhhg6k7l6y5reiwqr3noy Speedium network 245.34 TiB 4,493 LDN v3 multisig
f1z7jogzx4x42wtilzb4lu6iotlad5rptt2acbzpi Speedium network 5.91 TiB 9 LDN v3 multisig
f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq Speedium NetworkSpeedium Network 832.00 GiB 9 LDN # 77

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

1.25PiB

Id

9310b40a-68b7-45f9-b58a-786862ad7fec

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Last two approvers

flyworker & kernelogic

Rule to calculate the allocation request amount

80% of total dc amount requested

DataCap allocation requested

1.25PiB

Total DataCap granted for client so far

6.74PiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

-1970324836974591B

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
211756 49 2PiB 9.27 480.32TiB
filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

⚠️ 27.57% of total deal sealed by f01208803 are duplicate data.

⚠️ 28.09% of total deal sealed by f01157271 are duplicate data.

⚠️ 24.28% of total deal sealed by f01208154 are duplicate data.

⚠️ 27.18% of total deal sealed by f01208632 are duplicate data.

⚠️ 50.24% of total deal sealed by f01208189 are duplicate data.

⚠️ 24.42% of total deal sealed by f01156835 are duplicate data.

⚠️ 34.04% of total deal sealed by f01208042 are duplicate data.

⚠️ f01908299 has unknown IP location.

Provider Location Total Deals Sealed Percentage Unique Data Duplicate Deals
f01919423 Sydney, New South Wales, AU
Andrew Sjoquist Enterprises Pty Ltd
597.30 TiB 9.05% 576.67 TiB 3.45%
f01938357 Sydney, New South Wales, AU
Andrew Sjoquist Enterprises Pty Ltd
431.94 TiB 6.55% 431.91 TiB 0.01%
f01864434 Sydney, New South Wales, AU
Andrew Sjoquist Enterprises Pty Ltd
3.22 TiB 0.05% 3.22 TiB 0.00%
f01208803 Sydney, New South Wales, AU
Anycast Global Backbone
106.91 TiB 1.62% 77.44 TiB 27.57%
f01157271 Sydney, New South Wales, AU
Anycast Global Backbone
103.81 TiB 1.57% 74.66 TiB 28.09%
f01208154 Sydney, New South Wales, AU
Anycast Global Backbone
86.09 TiB 1.31% 65.19 TiB 24.28%
f01208632 Sydney, New South Wales, AU
Anycast Global Backbone
77.84 TiB 1.18% 56.69 TiB 27.18%
f01208189 Sydney, New South Wales, AU
Anycast Global Backbone
77.38 TiB 1.17% 38.50 TiB 50.24%
f01156835new Sydney, New South Wales, AU
Anycast Global Backbone
69.22 TiB 1.05% 52.31 TiB 24.42%
f01157249 Sydney, New South Wales, AU
Anycast Global Backbone
55.63 TiB 0.84% 49.72 TiB 10.62%
f01156883 Sydney, New South Wales, AU
Anycast Global Backbone
40.31 TiB 0.61% 38.22 TiB 5.19%
f01156538new Sydney, New South Wales, AU
Anycast Global Backbone
33.66 TiB 0.51% 27.47 TiB 18.38%
f01157027 Sydney, New South Wales, AU
Anycast Global Backbone
31.34 TiB 0.48% 25.28 TiB 19.34%
f01156975 Sydney, New South Wales, AU
Anycast Global Backbone
30.88 TiB 0.47% 25.78 TiB 16.50%
f01156901 Sydney, New South Wales, AU
Anycast Global Backbone
28.13 TiB 0.43% 26.53 TiB 5.67%
f01157018 Sydney, New South Wales, AU
Anycast Global Backbone
26.75 TiB 0.41% 23.34 TiB 12.73%
f01154023 Sydney, New South Wales, AU
Anycast Global Backbone
19.84 TiB 0.30% 18.63 TiB 6.14%
f01981059 Irving, Texas, US
AT&T Services, Inc.
202.56 TiB 3.07% 202.56 TiB 0.00%
f022352 Oslo, Oslo, NO
Blix Solutions AS
17.25 TiB 0.26% 17.06 TiB 1.09%
f01938665 Sham Shui Po, Sham Shui Po, HK
China Unicom Global
172.31 TiB 2.61% 170.25 TiB 1.20%
f01059489 Plano, Texas, US
Cogent Communications
147.78 TiB 2.24% 147.78 TiB 0.00%
f01996927new Plano, Texas, US
Cogent Communications
49.06 TiB 0.74% 49.06 TiB 0.00%
f01906874 Plano, Texas, US
Cogent Communications
2.50 TiB 0.04% 2.50 TiB 0.00%
f01207874 Sydney, New South Wales, AU
DATACOM SYSTEMS (AU) PTY LTD
62.69 TiB 0.95% 53.91 TiB 14.01%
f01208042 Sydney, New South Wales, AU
DATACOM SYSTEMS (AU) PTY LTD
56.38 TiB 0.85% 37.19 TiB 34.04%
f01207954 Sydney, New South Wales, AU
DATACOM SYSTEMS (AU) PTY LTD
45.97 TiB 0.70% 42.44 TiB 7.68%
f01206408 Sydney, New South Wales, AU
DATACOM SYSTEMS (AU) PTY LTD
22.13 TiB 0.34% 20.41 TiB 7.77%
f01992630 Louisville, Kentucky, US
Flexential Colorado Corp.
123.41 TiB 1.87% 123.41 TiB 0.00%
f01971600 Louisville, Kentucky, US
Flexential Colorado Corp.
104.75 TiB 1.59% 104.75 TiB 0.00%
f01878201 Hangzhou, Zhejiang, CN
Hangzhou Alibaba Advertising Co.,Ltd.
288.00 GiB 0.00% 288.00 GiB 0.00%
f01518369 San Jose, California, US
HONG KONG Megalayer Technology Co.,Limited
297.22 TiB 4.51% 297.22 TiB 0.00%
f01851482 Busan, Busan, KR
Korea Telecom
383.06 TiB 5.81% 383.06 TiB 0.00%
f0855584new Lincoln, Nebraska, US
LightEdge Solutions
50.03 TiB 0.76% 50.03 TiB 0.00%
f01091851 Lincoln, Nebraska, US
LightEdge Solutions
18.28 TiB 0.28% 18.28 TiB 0.00%
f01736668 Lincoln, Nebraska, US
LightEdge Solutions
17.63 TiB 0.27% 17.63 TiB 0.00%
f01882184 Herndon, Virginia, US
PCCW Global, Inc.
296.03 TiB 4.49% 296.03 TiB 0.00%
f01880047 Herndon, Virginia, US
PCCW Global, Inc.
259.88 TiB 3.94% 259.88 TiB 0.00%
f01877571 Herndon, Virginia, US
PCCW Global, Inc.
253.53 TiB 3.84% 253.38 TiB 0.06%
f01878005 Herndon, Virginia, US
PCCW Global, Inc.
146.69 TiB 2.22% 146.19 TiB 0.34%
f01882177 Herndon, Virginia, US
PCCW Global, Inc.
142.47 TiB 2.16% 142.41 TiB 0.04%
f01852664 Singapore, Singapore, SG
StarHub Ltd
306.91 TiB 4.65% 306.91 TiB 0.00%
f01938717 Singapore, Singapore, SG
StarHub Ltd
164.16 TiB 2.49% 163.94 TiB 0.13%
f01908671 New York City, New York, US
Unitas Global LLC
503.09 TiB 7.63% 460.88 TiB 8.39%
f01967501new New York City, New York, US
Unitas Global LLC
461.16 TiB 6.99% 461.16 TiB 0.00%
f01720359 Heerhugowaard, North Holland, NL
Wijnand Schouten trading as Speedium
184.78 TiB 2.80% 179.19 TiB 3.03%
f01786387 Heerhugowaard, North Holland, NL
Wijnand Schouten trading as Speedium
114.28 TiB 1.73% 110.69 TiB 3.14%
f01771403 Heerhugowaard, North Holland, NL
Wijnand Schouten trading as Speedium
96.50 TiB 1.46% 89.53 TiB 7.22%
f01199430 Heerhugowaard, North Holland, NL
Wijnand Schouten trading as Speedium
27.09 TiB 0.41% 24.44 TiB 9.80%
f01937642new Heerhugowaard, North Holland, NL
Wijnand Schouten trading as Speedium
26.66 TiB 0.40% 26.00 TiB 2.46%
f01908299 Unknown
Unknown
20.44 TiB 0.31% 20.38 TiB 0.31%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

✔️ Data replication looks healthy.

Unique Data Size Total Deals Made Number of Providers Deal Percentage
12.39 TiB 12.45 TiB 1 0.19%
69.50 TiB 143.66 TiB 2 2.18%
167.03 TiB 528.06 TiB 3 8.00%
203.91 TiB 861.66 TiB 4 13.06%
184.19 TiB 972.59 TiB 5 14.74%
167.63 TiB 1.02 PiB 6 15.88%
128.94 TiB 943.94 TiB 7 14.31%
106.72 TiB 892.56 TiB 8 13.53%
65.69 TiB 621.84 TiB 9 9.43%
34.66 TiB 362.09 TiB 10 5.49%
14.25 TiB 163.91 TiB 11 2.48%
3.22 TiB 40.28 TiB 12 0.61%
480.00 GiB 6.25 TiB 13 0.09%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

However, this could be possible if all below clients use same software to prepare for the exact same dataset or they belong to a series of LDN applications for the same dataset.

⚠️ CID sharing has been observed.

Other Client Application Total Deals Affected Unique CIDs Approvers
f1ws3n5tuxtyg26lraqkjirz7qon7y7ckju7hhmii Tech Greedy Inc 4.65 PiB 13,100 5cryptowhizzard
4Fenbushi-Filecoin
1IreneYoung
1jamerduhgamer
6kernelogic
1MegTei
1s0nik42
3xingjitansuo
f1pc5usvsbfgxxbq7c7quhhg6k7l6y5reiwqr3noy Speedium network 245.34 TiB 4,493 3kernelogic
1liyunzhi-666
1MegTei
1neogeweb3
2Reiers
3s0nik42
1xingjitansuo
f1z7jogzx4x42wtilzb4lu6iotlad5rptt2acbzpi Speedium network 6.19 TiB 9 1flyworker
1kernelogic
4MegTei
2psh0691
3Reiers
3s0nik42
f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq Speedium NetworkSpeedium Network 832.00 GiB 9 2dannyob
3flyworker
4MegTei
3Reiers
6s0nik42

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

Kevin-FF-USA commented 1 year ago

Datacap Request Trigger

Total DataCap requested

1 PiB

Expected weekly DataCap usage rate

400 TiB

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

DataCap allocation requested

256TiB

Id

feee3cd5-ac51-47fa-9810-cf45b5f1223a

NiwanDao commented 1 year ago

@cryptowhizzard The distribution looks healthy, but can you please explain the CID sharing with other entity https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/335 ?

cryptowhizzard commented 1 year ago

@xingjitansuo

Yes i can. #335 and we did build exactly the same dataset from AWS S3. This means the same CID's will be created.

NiwanDao commented 1 year ago

The same CID only happens when the Data preparer uses the same way on the same raw data to prepare the car file.

cryptowhizzard commented 1 year ago

@xingjitansuo yes , we did. We build it with Singularity, so did Xinan.

NiwanDao commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceapnxsq4z54ed4fxvihz4bjuy2dhycfz4sdkqtolw4exp5x3hcyeu

Address

f1xafi3wsh7q553efnumhgeomathrehwwd2mye7oy

Datacap Allocated

256.00TiB

Signer Address

f1a2lia2cwwekeubwo4nppt4v4vebxs2frozarz3q

Id

feee3cd5-ac51-47fa-9810-cf45b5f1223a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceapnxsq4z54ed4fxvihz4bjuy2dhycfz4sdkqtolw4exp5x3hcyeu

xinaxu commented 1 year ago

checker:manualTrigger f1ws3n5tuxtyg26lraqkjirz7qon7y7ckju7hhmii f1pc5usvsbfgxxbq7c7quhhg6k7l6y5reiwqr3noy