search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] Speedium - NexRad V2 [ Replica distribution ] Part 1 / 5 #483

Closed cryptowhizzard closed 1 year ago

cryptowhizzard commented 2 years ago

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Short addendum on this project

This application is a followup of our first. NexRad is 2 PiB +. We are building the dataset at the moment. This datacap is solely for replica distribution. Link -> https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/340

Share a brief history of your project and organization.

Speedium / Dcent has engaged in Slingshot starting 2.6. We have successfully stored more than 15 differerent datasets with 20+ different miners.

What is the primary source of funding for this project?

Company account

What other projects/ecosystem stakeholders is this project associated with?

Slingshot competition hosted by Protocol Labs

Use-case details

Describe the data being stored onto Filecoin

The [Next Generation Weather Radar](https://www.ncdc.noaa.gov/data-access/radar-data/nexrad) (NEXRAD) is a network of 160 high-resolution Doppler radar sites that detects precipitation and atmospheric movement and disseminates data in approximately 5 minute intervals from each site.

Where was the data in this dataset sourced from?

AWS

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this. 

https://www.ncei.noaa.gov/products/radar-data/nexrad

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

Yes.

What is the expected retrieval frequency for this data?

Multiple times p/y

For how long do you plan to keep this dataset stored on Filecoin?

18 months or longer

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

EU / US / Australia

How will you be distributing your data to storage providers? Is there an offline data transfer process?

The data will be transferred both offline and online.

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

We have a few providers who have been working with us during Slingshot Restore program and we'd like to continue working with them for ongoing Slingshot competition.

How will you be distributing deals across storage providers?

Max 2 copy's per storage provider if stored on different miners / locations.

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

Yes, we have the resources.
filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

Other Addresses[^2]

Storage Provider Distribution

The below table shows the distribution of storage providers that have stored data for this client.

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

⚠️ 26.75% of total deal sealed by f01208803 are duplicate data.

⚠️ 28.09% of total deal sealed by f01157271 are duplicate data.

⚠️ 24.21% of total deal sealed by f01208154 are duplicate data.

⚠️ 27.18% of total deal sealed by f01208632 are duplicate data.

⚠️ 50.24% of total deal sealed by f01208189 are duplicate data.

⚠️ 24.42% of total deal sealed by f01156835 are duplicate data.

⚠️ 34.04% of total deal sealed by f01208042 are duplicate data.

⚠️ 33.65% of total deal sealed by f01908671 are duplicate data.

Provider Location Total Deals Sealed Percentage Unique Data Duplicate Deals
f01919423 Sydney, New South Wales, AU
Andrew Sjoquist Enterprises Pty Ltd		 813.30 TiB 5.35% 712.23 TiB 12.43%
f01938357 Sydney, New South Wales, AU
Andrew Sjoquist Enterprises Pty Ltd		 538.78 TiB 3.54% 519.94 TiB 3.50%
f01864434 Sydney, New South Wales, AU
Andrew Sjoquist Enterprises Pty Ltd		 206.53 TiB 1.36% 202.13 TiB 2.13%
f01208803 Melbourne, Victoria, AU
Anycast Global Backbone		 111.69 TiB 0.73% 81.81 TiB 26.75%
f01157271 Melbourne, Victoria, AU
Anycast Global Backbone		 103.81 TiB 0.68% 74.66 TiB 28.09%
f01208154 Melbourne, Victoria, AU
Anycast Global Backbone		 86.34 TiB 0.57% 65.44 TiB 24.21%
f01208632 Melbourne, Victoria, AU
Anycast Global Backbone		 77.84 TiB 0.51% 56.69 TiB 27.18%
f01208189 Melbourne, Victoria, AU
Anycast Global Backbone		 77.38 TiB 0.51% 38.50 TiB 50.24%
f01156835new Melbourne, Victoria, AU
Anycast Global Backbone		 69.22 TiB 0.46% 52.31 TiB 24.42%
f01157249 Melbourne, Victoria, AU
Anycast Global Backbone		 56.19 TiB 0.37% 50.22 TiB 10.62%
f01156883 Melbourne, Victoria, AU
Anycast Global Backbone		 40.31 TiB 0.27% 38.22 TiB 5.19%
f01156538new Melbourne, Victoria, AU
Anycast Global Backbone		 34.38 TiB 0.23% 28.19 TiB 18.00%
f01157027 Melbourne, Victoria, AU
Anycast Global Backbone		 31.34 TiB 0.21% 25.28 TiB 19.34%
f01156975 Melbourne, Victoria, AU
Anycast Global Backbone		 30.88 TiB 0.20% 25.78 TiB 16.50%
f01156901 Melbourne, Victoria, AU
Anycast Global Backbone		 28.13 TiB 0.18% 26.53 TiB 5.67%
f01157018 Melbourne, Victoria, AU
Anycast Global Backbone		 26.75 TiB 0.18% 23.34 TiB 12.73%
f01154023 Melbourne, Victoria, AU
Anycast Global Backbone		 19.84 TiB 0.13% 18.63 TiB 6.14%
f01981059new Dallas, Texas, US
AT&T Services, Inc.		 404.53 TiB 2.66% 404.53 TiB 0.00%
f022352 Oslo, Oslo, NO
Blix Solutions AS		 17.25 TiB 0.11% 17.06 TiB 1.09%
f010479 Paris, Île-de-France, FR
CELESTE SAS		 41.44 TiB 0.27% 37.34 TiB 9.88%
f01962819 Xi’an, Shaanxi, CN
China Telecom (Group)		 97.16 TiB 0.64% 97.16 TiB 0.00%
f0750779new Chengdu, Sichuan, CN
CHINA UNICOM China169 Backbone		 302.56 TiB 1.99% 301.94 TiB 0.21%
f01938665 Sham Shui Po, Sham Shui Po, HK
China Unicom Global		 172.31 TiB 1.13% 170.25 TiB 1.20%
f039940 Chengdu, Sichuan, CN
CHINANET-BACKBONE		 142.59 TiB 0.94% 142.41 TiB 0.13%
f01901765 Jiaxing, Zhejiang, CN
CHINANET-BACKBONE		 90.91 TiB 0.60% 90.91 TiB 0.00%
f01902461new Zhengzhou, Henan, CN
CHINANET-BACKBONE		 64.69 TiB 0.43% 64.69 TiB 0.00%
f01059489 Plano, Texas, US
Cogent Communications		 243.66 TiB 1.60% 239.84 TiB 1.56%
f01996927new Plano, Texas, US
Cogent Communications		 215.34 TiB 1.42% 215.34 TiB 0.00%
f02026730new Antwerpen, Flanders, BE
Cogent Communications		 87.06 TiB 0.57% 86.47 TiB 0.68%
f01345523 Antwerpen, Flanders, BE
Cogent Communications		 80.38 TiB 0.53% 80.06 TiB 0.39%
f01906874 Plano, Texas, US
Cogent Communications		 63.09 TiB 0.41% 63.09 TiB 0.00%
f02012223 Antwerpen, Flanders, BE
Cogent Communications		 24.84 TiB 0.16% 24.84 TiB 0.00%
f01858429 Boston, Massachusetts, US
Comcast Cable Communications, LLC		 65.59 TiB 0.43% 65.59 TiB 0.00%
f09848 Rancho Santa Margarita, California, US
Cox Communications Inc.		 190.94 TiB 1.26% 190.25 TiB 0.36%
f01761579new Hangzhou, Zhejiang, CN
CT-HangZhou-IDC		 380.69 TiB 2.50% 378.75 TiB 0.51%
f01207874 Sydney, New South Wales, AU
DATACOM SYSTEMS (AU) PTY LTD		 62.69 TiB 0.41% 53.91 TiB 14.01%
f01208042 Sydney, New South Wales, AU
DATACOM SYSTEMS (AU) PTY LTD		 56.38 TiB 0.37% 37.19 TiB 34.04%
f01207954 Sydney, New South Wales, AU
DATACOM SYSTEMS (AU) PTY LTD		 45.97 TiB 0.30% 42.44 TiB 7.68%
f01206408 Sydney, New South Wales, AU
DATACOM SYSTEMS (AU) PTY LTD		 22.13 TiB 0.15% 20.41 TiB 7.77%
f01971600 Dallas, Texas, US
Flexential Colorado Corp.		 524.88 TiB 3.45% 524.88 TiB 0.00%
f01992630 Dallas, Texas, US
Flexential Colorado Corp.		 524.28 TiB 3.45% 524.28 TiB 0.00%
f01920723 Maywood Park, Oregon, US
Flexential Colorado Corp.		 414.94 TiB 2.73% 407.16 TiB 1.88%
f01988794 Dallas, Texas, US
Flexential Colorado Corp.		 124.66 TiB 0.82% 124.66 TiB 0.00%
f01985775 Dallas, Texas, US
Flexential Colorado Corp.		 100.58 TiB 0.66% 100.58 TiB 0.00%
f01985745 Dallas, Texas, US
Flexential Colorado Corp.		 79.66 TiB 0.52% 79.66 TiB 0.00%
f01392893 Amsterdam, North Holland, NL
Fusix Networks B.V.		 121.06 TiB 0.80% 121.06 TiB 0.00%
f01878201 Hangzhou, Zhejiang, CN
Hangzhou Alibaba Advertising Co.,Ltd.		 98.84 TiB 0.65% 98.84 TiB 0.00%
f01911083 Tuusula, Uusimaa, FI
Hetzner Online GmbH		 99.75 TiB 0.66% 99.75 TiB 0.00%
f01894158new Hong Kong, Central and Western, HK
HK Broadband Network Ltd.		 204.34 TiB 1.34% 204.19 TiB 0.08%
f01907545 Hong Kong, Central and Western, HK
HK Broadband Network Ltd.		 59.00 TiB 0.39% 59.00 TiB 0.00%
f01518369 San Jose, California, US
HONG KONG Megalayer Technology Co.,Limited		 297.22 TiB 1.95% 297.22 TiB 0.00%
f01896422 Fremont, California, US
Hurricane Electric LLC		 99.72 TiB 0.66% 99.72 TiB 0.00%
f01926686 Hangzhou, Zhejiang, CN
Jiangxi Jiujiang IDC		 98.22 TiB 0.65% 98.22 TiB 0.00%
f01851482 Busan, Busan, KR
Korea Telecom		 460.91 TiB 3.03% 460.91 TiB 0.00%
f01736668 Lincoln, Nebraska, US
LightEdge Solutions		 143.63 TiB 0.94% 140.72 TiB 2.02%
f0855584new Lincoln, Nebraska, US
LightEdge Solutions		 94.00 TiB 0.62% 90.94 TiB 3.26%
f01091851 Lincoln, Nebraska, US
LightEdge Solutions		 30.41 TiB 0.20% 30.41 TiB 0.00%
f01908299 Lincoln, Nebraska, US
LightEdge Solutions		 20.44 TiB 0.13% 20.38 TiB 0.31%
f01775922 Ashburn, Virginia, US
New Dream Network, LLC		 24.72 TiB 0.16% 24.69 TiB 0.13%
f01652333 Sunnyvale, California, US
Packet Host, Inc.		 351.38 TiB 2.31% 346.09 TiB 1.50%
f01662356 Ashburn, Virginia, US
Packet Host, Inc.		 64.00 GiB 0.00% 64.00 GiB 0.00%
f01882184 Singapore, Singapore, SG
PCCW Global, Inc.		 392.13 TiB 2.58% 392.13 TiB 0.00%
f01880047 Singapore, Singapore, SG
PCCW Global, Inc.		 259.88 TiB 1.71% 259.88 TiB 0.00%
f01877571 Singapore, Singapore, SG
PCCW Global, Inc.		 253.53 TiB 1.67% 253.38 TiB 0.06%
f01878005 Singapore, Singapore, SG
PCCW Global, Inc.		 146.69 TiB 0.96% 146.19 TiB 0.34%
f01882177 Singapore, Singapore, SG
PCCW Global, Inc.		 142.47 TiB 0.94% 142.41 TiB 0.04%
f01873432 Las Vegas, Nevada, US
PiKNiK & Company Inc.		 332.09 TiB 2.18% 328.47 TiB 1.09%
f01851060new Las Vegas, Nevada, US
PiKNiK & Company Inc.		 319.53 TiB 2.10% 314.16 TiB 1.68%
f01904630 Las Vegas, Nevada, US
PiKNiK & Company Inc.		 140.31 TiB 0.92% 138.13 TiB 1.56%
f01419959 San Diego, California, US
ScaleMatrix		 36.50 TiB 0.24% 35.53 TiB 2.65%
f01863339 Shenzhen, Guangdong, CN
Shenzhen Tencent Computer Systems Company Limited		 70.22 TiB 0.46% 68.44 TiB 2.54%
f01852664 Singapore, Singapore, SG
StarHub Ltd		 344.28 TiB 2.26% 344.28 TiB 0.00%
f01938717 Singapore, Singapore, SG
StarHub Ltd		 164.16 TiB 1.08% 163.94 TiB 0.13%
f01850141 Hong Kong, Central and Western, HK
Tencent Building, Kejizhongyi Avenue		 314.59 TiB 2.07% 313.19 TiB 0.45%
f01908671 Atlanta, Georgia, US
Unitas Global LLC		 778.38 TiB 5.12% 516.47 TiB 33.65%
f01967501new Atlanta, Georgia, US
Unitas Global LLC		 512.81 TiB 3.37% 512.81 TiB 0.00%
f01119939 Atlanta, Georgia, US
Unitas Global LLC		 263.34 TiB 1.73% 260.94 TiB 0.91%
f01901739 Sydney, New South Wales, AU
Vocus Connect International Backbone		 108.09 TiB 0.71% 95.31 TiB 11.82%
f01907460 Seattle, Washington, US
Wholesail networks LLC		 591.44 TiB 3.89% 591.44 TiB 0.00%
f0838923 Seattle, Washington, US
Wholesail networks LLC		 274.41 TiB 1.80% 271.19 TiB 1.17%
f01851231 Seattle, Washington, US
Wholesail networks LLC		 21.25 TiB 0.14% 21.25 TiB 0.00%
f01720359 Heerhugowaard, North Holland, NL
Wijnand Schouten trading as Speedium		 230.00 TiB 1.51% 224.34 TiB 2.46%
f01786387 Heerhugowaard, North Holland, NL
Wijnand Schouten trading as Speedium		 138.41 TiB 0.91% 134.81 TiB 2.60%
f01771403 Heerhugowaard, North Holland, NL
Wijnand Schouten trading as Speedium		 124.56 TiB 0.82% 116.69 TiB 6.32%
f01199430 Heerhugowaard, North Holland, NL
Wijnand Schouten trading as Speedium		 27.09 TiB 0.18% 24.44 TiB 9.80%
f01937642new Heerhugowaard, North Holland, NL
Wijnand Schouten trading as Speedium		 26.66 TiB 0.18% 26.00 TiB 2.46%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

✔️ Data replication looks healthy.

Unique Data Size Total Deals Made Number of Providers Deal Percentage
594.66 TiB 594.69 TiB 1 3.91%
19.55 TiB 39.91 TiB 2 0.26%
64.38 TiB 199.78 TiB 3 1.31%
147.56 TiB 608.81 TiB 4 4.00%
183.44 TiB 958.63 TiB 5 6.30%
126.88 TiB 788.31 TiB 6 5.18%
105.19 TiB 758.69 TiB 7 4.99%
58.25 TiB 484.47 TiB 8 3.19%
42.47 TiB 395.31 TiB 9 2.60%
24.41 TiB 260.22 TiB 10 1.71%
21.59 TiB 258.91 TiB 11 1.70%
20.03 TiB 260.25 TiB 12 1.71%
24.28 TiB 338.50 TiB 13 2.23%
29.75 TiB 450.16 TiB 14 2.96%
42.13 TiB 664.09 TiB 15 4.37%
36.75 TiB 621.25 TiB 16 4.08%
34.38 TiB 621.19 TiB 17 4.08%
31.97 TiB 608.72 TiB 18 4.00%
17.38 TiB 361.06 TiB 19 2.37%
14.78 TiB 321.44 TiB 20 2.11%
17.31 TiB 388.09 TiB 21 2.55%
20.03 TiB 461.72 TiB 22 3.04%
23.63 TiB 568.78 TiB 23 3.74%
24.53 TiB 612.59 TiB 24 4.03%
24.38 TiB 635.03 TiB 25 4.17%
25.53 TiB 690.78 TiB 26 4.54%
22.88 TiB 642.97 TiB 27 4.23%
17.75 TiB 518.28 TiB 28 3.41%
12.63 TiB 381.38 TiB 29 2.51%
10.94 TiB 342.69 TiB 30 2.25%
6.97 TiB 227.19 TiB 31 1.49%
3.00 TiB 100.53 TiB 32 0.66%
1.22 TiB 42.47 TiB 33 0.28%
96.00 GiB 3.50 TiB 34 0.02%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

However, this could be possible if all below clients use same software to prepare for the exact same dataset or they belong to a series of LDN applications for the same dataset.[^3]

⚠️ CID sharing has been observed.

Other Client Application Total Deals Affected Unique CIDs Approvers
f1z7jogzx4x42wtilzb4lu6iotlad5rptt2acbzpi Speedium network 235.94 TiB 3,212 1flyworker
1kernelogic
4MegTei
2psh0691
3Reiers
3s0nik42
f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq Speedium NetworkSpeedium Network 832.00 GiB 9 2dannyob
3flyworker
4MegTei
3Reiers
6s0nik42
f1klrehuwhzfw6boiiywdi7pikztsob4kkieivcwq InternationalGenome (EMBL-EBI) 480.00 GiB 2 1kernelogic
1mjroddy
1psh0691
1s0nik42
1xinaxu
1xingjitansuo

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

data-programs commented 1 year ago
KYC

This user’s identity has been verified through filplus.storage