search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] NDLABS - Hyper.ai <3/4> #1722

Closed NDLABS-Leo closed 1 year ago

NDLABS-Leo commented 1 year ago

Data Owner Name

NDLABS

Data Owner Country/Region

Singapore

Data Owner Industry

IT & Technology Services

Website

https://www.ndlabs.io/#/

Social Media

Twitter： @imNDLABS
Slack： @NDLABS-OFFICE

Total amount of DataCap being requested

5PiB

Weekly allocation of DataCap requested

1PiB

On-chain address for first allocation

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Custom multisig

Identifier

No response

Share a brief history of your project and organization

ND LABS has technical operation centers and nodes in Singapore, Hong Kong, the United States, and Dubai. Since Fil has launch of the mainnet in 2020, ND has begun to provide technical services to partners to help them complete the construction of storage services. At present, the accumulated storage power of ND exceeds 300P globally. The largest node has 100P storage power, and the node owns exceeds more than 1.4 million FIL. 
ND LABS is positioned as a decentralized storage service provider for WEB3. For a long time, ND not only focuses on building nodes for partners, but also explores how to provide better storage services for potential clients of web3. Since October 2021, ND has been deeply involved in the FilPlus project, vigorously promoting the Filplus project to partners who has effective data storage needs. We also providing them with a complete set of solutions and technical services for storing data in the FIL network. The Singapore and US nodes are the main storage nodes, which was provide real data storage for early customers.

Is this project associated with other projects/ecosystem stakeholders?

Yes

If answered yes, what are the other projects/ecosystem stakeholders

We gonna cooperate with more SPs from other regions

Describe the data being stored onto Filecoin

https://hyper.ai/datasets is an open data web, which collects hundreds of datasets with a total capacity of 1.8P for public use.
Details of Datasets including:
Audi Autonomous Driving Dataset - 2.26T
BDD100K - 1.81T
BDDK - 1.8T
LSUN 20 Object Categories - 1.69T
Youtube 8M - 1.52T
WebVision1.0+2.0 - 1.26T
ApolloSpace - 1.19T
TrackingNet - 1.04T
...
FYI, to release the concern from community members of "one LDN combined smaller sets of data". 
When we prepare the data, different datasets will be tagged variously, to classify into our database. If we need to index later, we can quickly and easily find the file corresponding to the demand. In addition, we are also developing a browser to solve the "merged data"

Where was the data currently stored in this dataset sourced from

Other

If you answered "Other" in the previous question, enter the details here

https://hyper.ai/datasets

How do you plan to prepare the dataset

IPFS, lotus

If you answered "other/custom tool" in the previous question, enter the details here

No response

Please share a sample of the data

https://hyper.ai/datasets/15610
https://hyper.ai/datasets/11926
https://hyper.ai/datasets/5570
https://hyper.ai/datasets/5678
https://hyper.ai/datasets/5531
https://hyper.ai/datasets/16979
https://hyper.ai/datasets/5197
https://hyper.ai/datasets/16773
https://hyper.ai/datasets/8754
https://hyper.ai/datasets/4889
https://hyper.ai/datasets/17858
https://hyper.ai/datasets/16184
https://hyper.ai/datasets/5191
https://hyper.ai/datasets/5358
https://hyper.ai/datasets/9430

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

1.5 to 2 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, North America

How will you be distributing your data to storage providers

HTTP or FTP server, IPFS, Shipping hard drives

How do you plan to choose storage providers

Slack, Big data exchange, Partners

If you answered "Others" in the previous question, what is the tool or platform you plan to use

No response

If you already have a list of storage providers to work with, fill out their names and provider IDs below

No response

How do you plan to make deals to your storage providers

Boost client, Lotus client

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

Sunnyiscoming commented 1 year ago

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

1PiB

Client address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

DataCap allocation requested

256TiB

Id

06fe0afc-69df-416b-b691-d355ef523ec6

kernelogic commented 1 year ago

I have checked https://hyper.ai/datasets and looks like it's very comprehensive for dataset indexing. Willing to support.

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedb3wwp5nocyhbhs7hhsdgh3tfhvcpsoecehpjlq5jugl4rryhex6

Address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Datacap Allocated

256.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

06fe0afc-69df-416b-b691-d355ef523ec6

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedb3wwp5nocyhbhs7hhsdgh3tfhvcpsoecehpjlq5jugl4rryhex6

sxxfuture-official commented 1 year ago

Base on the answer in #1720 and By checking the public big data set link provided, the information is true and reliable, and ND is not a new account, so I will support the project in this round.

sxxfuture-official commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedyodhoh6ipm446mhp56b2kws4j7puea7mxpt4oz2zadruc4tdpuu

Address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Datacap Allocated

256.00TiB

Signer Address

f1foiomqlmoshpuxm6aie4xysffqezkjnokgwcecq

Id

06fe0afc-69df-416b-b691-d355ef523ec6

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedyodhoh6ipm446mhp56b2kws4j7puea7mxpt4oz2zadruc4tdpuu

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

DataCap allocation requested

512TiB

Id

c1d04631-f64a-47b1-9a77-92c52566749e

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Rule to calculate the allocation request amount

10% of total dc amount requested

DataCap allocation requested

512TiB

Total DataCap granted for client so far

256TiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

4.75PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
6065 6 256TiB 21.1 59.31TiB
laurarenpanda commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceddxow62ntjvblb4rw22lrca7q4vrqkpc7qcgmeyb6isc5eynvowm

Address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Datacap Allocated

512.00TiB

Signer Address

f1bp3tzp536edm7dodldceekzbsx7zcy7hdfg6uzq

Id

c1d04631-f64a-47b1-9a77-92c52566749e

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceddxow62ntjvblb4rw22lrca7q4vrqkpc7qcgmeyb6isc5eynvowm

DaYouGroup commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 100.00% of deals are for data replicated across less than 3 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

DaYouGroup commented 1 year ago

NDLABS is reputable and we are confident that you can solve this problem and are willing to support this round.

DaYouGroup commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacectuw7uvctltrizpg22udu6zz5j6e3jfio5fjlv32nptnmlxemzhk

Address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Datacap Allocated

512.00TiB

Signer Address

f1nwjsd2mc6hu4qrwnmd6ukrfkuu4h5fhs7u3exii

Id

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacectuw7uvctltrizpg22udu6zz5j6e3jfio5fjlv32nptnmlxemzhk

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

DataCap allocation requested

1PiB

Id

2839901a-5f0a-43cb-a34b-58e8cea24816

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Rule to calculate the allocation request amount

20% of total dc amount requested

DataCap allocation requested

1PiB

Total DataCap granted for client so far

465661.3YiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

-5.62B

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
20342 12 512TiB 22.81 130.03TiB
Chuangshi1 commented 1 year ago

It has been checked in https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1720

Chuangshi1 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceatltj2jurdna5hf5u25p6d34fnfkfiokgiqgo2cuyjp77bnv52hm

Address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Datacap Allocated

1.00PiB

Signer Address

f1mdk7s2vntzm6hu35yuo6vjubtrpfnb2awhgvrri

Id

2839901a-5f0a-43cb-a34b-58e8cea24816

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceatltj2jurdna5hf5u25p6d34fnfkfiokgiqgo2cuyjp77bnv52hm

zcfil commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 95.35% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

YuanHeHK commented 1 year ago

1720~#1723 are a series of applications of NDLABS, the situation is similar, based on the survey of 1720 I support them this time.

YuanHeHK commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecovreou5gfsz3sy2zc5ggbubtwwxhmdzzqgzifsia6lg3yxi6cpu

Address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Datacap Allocated

1.00PiB

Signer Address

f1fg6jkxsr3twfnyhdlatmq36xca6sshptscds7xa

Id

2839901a-5f0a-43cb-a34b-58e8cea24816

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecovreou5gfsz3sy2zc5ggbubtwwxhmdzzqgzifsia6lg3yxi6cpu

NDLABS-Leo commented 1 year ago

Hi community members, we have communicated the rules to our partner sp's as soon as the search check came out, we are still in ongoing communication with them about the reasons why different sp's are not retrieving data in a timely manner, at the moment our search rate is not high but not 0 retrievable. Please give us some time, we plan to improve the retrieval report within 2 weeks.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f02049625

Client address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

DataCap allocation requested

2PiB

Id

6cfb827a-1f36-4a77-ab13-1ac8341d4a12

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Rule to calculate the allocation request amount

400% weekly > 2PiB, requesting 2PiB

DataCap allocation requested

2PiB

Total DataCap granted for client so far

931322574615478927360.0YiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

931322574615478927360.0YiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
48791 17 1PiB 20.93 254.75TiB
filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 98.02% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

mikezli commented 1 year ago

Leo contacted me and promised to improve the retrieve success rate. Considering that they are well-known community members, I am willing to support them this round and keep paying attention.

mikezli commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceawteyfzeah73in4jdibekdclnstjkbjnjal4uf4iinxe6nkdpqlg

Address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Datacap Allocated

2.00PiB

Signer Address

f1dnb3uz7sylxk6emti3ififcvu3nlufnnsjui6ea

Id

6cfb827a-1f36-4a77-ab13-1ac8341d4a12

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceawteyfzeah73in4jdibekdclnstjkbjnjal4uf4iinxe6nkdpqlg

NDLABS-Leo commented 1 year ago

@mikezli Yes, we stored the unseal file of the data, but the node configuration problem it was not connected, currently it is being adjusted, it will be taken care of soon。Thank you for your support

luobin544 commented 1 year ago

WeChat22237c987f7f56ba3fa84c793496cd93

luobin544 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedhqvx5e5clsocf4hjicixcqxpcilhg6m3pyok4vmm2pawtawjbue

Address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Datacap Allocated

2.00PiB

Signer Address

f1tbd632f6w62glfaf7wjpimacbnjiz26poyoes2q

Id

6cfb827a-1f36-4a77-ab13-1ac8341d4a12

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedhqvx5e5clsocf4hjicixcqxpcilhg6m3pyok4vmm2pawtawjbue

1ane-1 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecxzhnqrikxugfeaaw3mxpww6v5hb2nmuppp4yifu2y6wm4i763aw

Address

f1dolkq3bxvdxsvcmdddfb7n62snciur4q2wtmrja

Datacap Allocated

2.00PiB

Signer Address

f1mdk7s2vntzm6hu35yuo6vjubtrpfnb2awhgvrri

Id

6cfb827a-1f36-4a77-ab13-1ac8341d4a12

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecxzhnqrikxugfeaaw3mxpww6v5hb2nmuppp4yifu2y6wm4i763aw

NDLABS-Leo commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 98.49% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

NDLABS-Leo commented 1 year ago

FYI, we are disclosing here for tracking by notaries. We have located that the reason why ND and the SPs we cooperate with cannot be searched and sampled by Filplus RetrievalBot is that our code is not synchronized with the latest boost code, so the payloadCID (rootCID) cannot be retrieved. However, pieceCID can be retrieved, which can be seen in previous notaries' reviewing process. ND and the SPs we cooperate with store 100% unseal files, from the beginning to the present. Our engineers are simultaneously modifying the code and updating the architecture, which is expected to be completed early this week. Looking forward to your understanding and signing. Thanks a lot.

Irmagogo commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 99.01% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

NDLABS-Leo commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 99.05% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

NDLABS-Leo commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 99.05% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

NDLABS-Leo commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 99.06% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

NDLABS-Leo commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

⚠️ 99.11% of deals are for data replicated across less than 4 storage providers.

Deal Data Shared with other Clients[^3]

⚠️ CID sharing has been observed. (Top 3)

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

data-programs commented 1 year ago
KYC

This user’s identity has been verified through filplus.storage

aggregation-and-compliance-bot[bot] commented 10 months ago
Client f02057947 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 7 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Cid Checker score > 25% The client has a CID checker score of 2%. This should be greater than 25%. To find out more about CID checker score please look at this issue: https://github.com/filecoin-project/notary-governance/issues/986