search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
109 stars 62 forks source link

[DataCap Application] Nebula Block - Common Crawl[4/10] #1532

Closed bq1024 closed 11 months ago

bq1024 commented 1 year ago

Data Owner Name

Nebula Block

Data Owner Country/Region

Canada

Data Owner Industry

Web3 / Crypto

Website

https://www.nebulablock.com/

Social Media

https://twitter.com/nbfsdata 
https://www.linkedin.com/company/nebula-block/ 
https://medium.com/@nbdata

Total amount of DataCap being requested

5PiB

Weekly allocation of DataCap requested

100TiB

On-chain address for first allocation

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

Custom multisig

Identifier

No response

Share a brief history of your project and organization

As a data center services provider and blockchain practitioner, Nebula Block is one of the earliest participants in the Filecoin ecosystem in North America and has been contributing to the Filecoin network and ecosystem since 2020. We were selected as minerX fellow, participated in early-stage testing, participated in slingshot v2 and v3, helped improve louts and support with ecosystem projects like Filmine, Estuary, and FilSwan. Nebula Block is also in charge of this dataset storage and deal preparation in slingshot v3.

Currently, Nebula block is one of the biggest web3 hosting providers in North America, which works with 4 data centers across the world, and hosts more than 3000 servers with 700 PiB storage.

Nebula Block is seeking to onboard data to meet the high demand of FIL+ for itself and its partners.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

The Common Crawl corpus contains petabytes of data collected since 2008. It contains raw web page data, extracted metadata, and text extractions.

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

No response

How do you plan to prepare the dataset

singularity, others/custom tool

If you answered "other/custom tool" in the previous question, enter the details here

We will use multiple tools to prepare the dataset, such as singularity and go-swan-client. https://github.com/filswan/go-swan-client

Please share a sample of the data

https://download2.nbfsdata.com/CC-MAIN-2022-27-out/baga6ea4seaqpzzxy6qvb6f7mhazdycsgn6qocu6uz7gzsw7l2py7dgdjlk4f2ay.car

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Monthly

For how long do you plan to keep this dataset stored on Filecoin

2 to 3 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, Africa, North America, South America, Europe, Australia (continent), Antarctica

How will you be distributing your data to storage providers

HTTP or FTP server, IPFS

How do you plan to choose storage providers

Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

We will create auto-bid tasks on the FilSwan platform, and the platform will help up find suitable storage providers based on its reputation system (https://docs.filswan.com/filswan-platform/overview/reputation-system)

If you already have a list of storage providers to work with, fill out their names and provider IDs below

Some storage providers registered on the FilSwan platform and accepting auto-bid deals are as follows:
f010088
f02301
f08399
f047419
f0187709
f01163272
f01402814
f0240185
f0143858
f0408717
f01225882
f0717969
f01211558
f03223
f01801091
f0240456
f032824
f01931631
f01907556
f01877184
f02003005

How do you plan to make deals to your storage providers

Others/custom tool

If you answered "Others/custom tool" in the previous question, enter the details here

We will use go-swan-client to make deals. https://github.com/filswan/go-swan-client

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

simonkim0515 commented 1 year ago

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

100TiB

Client address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f01858410

Client address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

DataCap allocation requested

50TiB

Id

3c22517d-f24c-4b68-8c33-261745456a87

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report[^1]

There is no previous allocation for this issue.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacech6lhzo6h6lrd3w2phntujqeszffuocc7ol5tadqnp72jhlzacvw

Address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

Datacap Allocated

50.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

3c22517d-f24c-4b68-8c33-261745456a87

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacech6lhzo6h6lrd3w2phntujqeszffuocc7ol5tadqnp72jhlzacvw

cryptowhizzard commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebthjcxc4cq2ohee2h7bxivuhpgwfv4tthhr3p2lpfznqnfynw5du

Address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

Datacap Allocated

50.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

3c22517d-f24c-4b68-8c33-261745456a87

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebthjcxc4cq2ohee2h7bxivuhpgwfv4tthhr3p2lpfznqnfynw5du

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

DataCap allocation requested

100TiB

Id

51d91fe5-37b3-4974-b4c6-046b2708290e

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

Last two approvers

cryptowhizzard & kernelogic

Rule to calculate the allocation request amount

100% of weekly dc amount requested

DataCap allocation requested

100TiB

Total DataCap granted for client so far

50TiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

4.95PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
1109 13 50TiB 15.87 12.21TiB
filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

maxvint commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

woshidama323 commented 1 year ago

Looks Good for me Will help to approve this

woshidama323 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebn2mcowpnz624rice34tcpblvltcs2yugfz2r4glqq3xswuz4v22

Address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

Datacap Allocated

100.00TiB

Signer Address

f12tk3adljauwnd3hjbigpfxb7b7gdlj63p6afwtq

Id

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebn2mcowpnz624rice34tcpblvltcs2yugfz2r4glqq3xswuz4v22

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

Rule to calculate the allocation request amount

100% of weekly dc amount requested

DataCap allocation requested

100TiB

Total DataCap granted for client so far

50TiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

4.95PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
1434 17 50TiB 12.27 2.84TiB
filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

Rule to calculate the allocation request amount

100% of weekly dc amount requested

DataCap allocation requested

100TiB

Total DataCap granted for client so far

50TiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

4.95PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
1434 17 50TiB 12.27 2.84TiB
filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

igoovo commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebse6nf3ww6e5x42ys3vru4dfvrwgqnx5uugaignkxgis3k6x3ztk

Address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

Datacap Allocated

100.00TiB

Signer Address

f1shnsfayxqll77svffaxnjenms7bbbysbqcatrpy

Id

51d91fe5-37b3-4974-b4c6-046b2708290e

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebse6nf3ww6e5x42ys3vru4dfvrwgqnx5uugaignkxgis3k6x3ztk

igoovo commented 1 year ago

Early packaging, and robot inspection without violations, provide support.

zcfil commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

zcfil commented 1 year ago

The data is widely distributed and appears to be very healthy. This round of review has been passed, and we will continue to pay attention to it in the future

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

DataCap allocation requested

100TiB

Id

204be48e-8d9a-4048-b062-5f9b0f1e9463

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

Rule to calculate the allocation request amount

100% of weekly dc amount requested

DataCap allocation requested

100TiB

Total DataCap granted for client so far

50TiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

4.95PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
4357 19 50TiB 18.54 12.84TiB
Normalnoise commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

zcfil commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

zcfil commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceac6tnluenzztul5ff7efcximp25lyhffoyl4nqfp7z2vmw5twoh6

Address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

Datacap Allocated

100.00TiB

Signer Address

f1cjzbiy5xd4ehera4wmbz63pd5ku4oo7g52cldga

Id

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceac6tnluenzztul5ff7efcximp25lyhffoyl4nqfp7z2vmw5twoh6

nj-steve commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

nj-steve commented 1 year ago

everything looks good!

nj-steve commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceao3zsfgmrtedmbsz5h4rc53yibc6jdpt6xic6j2hdqxc27o5x2re

Address

f1idqck2hbviikb5xpjmh3l7cposc23igpnd35q7a

Datacap Allocated

100.00TiB

Signer Address

f1xx6555qijma7igpnjspyvdunc4vfxkawnpqy5ii

Id

204be48e-8d9a-4048-b062-5f9b0f1e9463

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceao3zsfgmrtedmbsz5h4rc53yibc6jdpt6xic6j2hdqxc27o5x2re

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

bq1024 commented 1 year ago

We'd like to keep this application open as we are currently working on onboarding more SPs to the project

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

bq1024 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

clriesco commented 1 year ago

Removed stale label and reopened issue :)

bq1024 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

bq1024 commented 1 year ago

checker:manualTrigger