Closed zhongdonglin closed 1 year ago
About customer data: First, we got in touch with the official members of the fil+ team and a large number of notaries, who have done detailed due diligence on our application. Secondly, our sample data and the specific amount of data have been publicly disclosed in the due diligence of the notary public in the early stage, and can be viewed in this application. Finally, using Filecoin for storage is mainly because I see the future of distributed storage and hope to participate in the Web3 world. About files and processing: We mainly use the offline mail storage server for data distribution, and the server configuration as the transmission carrier is: X10DRL-I motherboard, E5-2680v42, 480G SSD1, DDR4 32G2, dual-port 10GbE network card + optical module 10G, LSI 9311-8I 2, CRPS1600W*1 In order to ensure the security of data during transmission, the storage system adopts ZFS-RaidZ2 storage pool, which allows data not to be lost if less than two disks are damaged. We use the Singularity program for data cutting and packaging, and at the same time, we have customized and optimized it to improve efficiency. In order to ensure the compliance of data encapsulation, we have established a corresponding management program that will monitor the entire life cycle of each car file.
@zhongdonglin thank you for your detailled response, highly appreciated.
Can you clarify the following tho things to me?
100% of your data storage happend in the same region, which is against the Filecoin+ guidelines. How are you going to disperse the data to multiple regions?
Your data is described as "It belongs to the video class tutorial data. As an important material for the company's employee training, these data can improve the basic skills and abilities of employees to meet the company's business requirements in relevant directions.The content includes corporate management, financial management, tax and economic law, etc., which is closely related to the company's main business. There is no doubt that the content of these courses is rich and professional."
This is business data and does not belong on the Filecoin+ program. Please use regular deals or contact the FIL-E program.
If you need any assistance let me know, happy to help you.
We do not store data in the same region. Storage providers are distributed in Hong Kong and different prefecture-level cities in Guangdong Province. In the future, new SPs will be added in Hubei, Beijing and other places.
The reason for using FIL+ for storage is that this project supports the storage of all useful data, and our data is also valuable data. In addition, we have not paid attention to the FIL-E you mentioned, and we will learn more about it later.
@zhongdonglin with regio we mean the Asian region. You should also store some of the data on other continents world wide.
My collegue is right. Business data belongs in Fil-E , not in Fil+.
On a second note: All the data stored has been made non retrievable.
No notary should sign this application untill this is fixed. @psh0691 can you please cancel your propose message?
First, let's hear the applicant's explanation.
@psh0691
No , first you do due diligence, then hear the explanation... THEN you sign if there is still a need to.
@zhongdonglin
Beautifull story. Now for the real world:
Why should you store your data so urgently when it is not retrievable at all? All SP's chosen do not support retrieval, the data is inaccessible.
About the regions, the world is bigger then your spot in China. Asia to my knowledge includes:
It belongs to the video class tutorial data. As an important material for the company's employee training, these data can improve the basic skills and abilities of employees to meet the company's business requirements in relevant directions.The content includes corporate management, financial management, tax and economic law, etc., which is closely related to the company's main business. There is no doubt that the content of these courses is rich and professional.
And is business data and belongs in Fil-E.
@cryptowhizzard @herrehesse I have answered the storage area question many times, even on filplush-checker it is compliant, am I wrong to follow my promise? From the comments of multiple DCs, it can be seen that your questions have a strong subjective consciousness, and the motivation is incredible, which is very different from most notaries. Many statements have not been investigated and verified in detail, which is very important to the general public. It's not fair to support Fil+'s client. I don't think your approach is of any help to the ecological development of Filecoin, I hope the official team could notice the relevant personnel and related situations @raghavrmadya @Kevin-FF-USA
@zhongdonglin
I understand that it might be difficult to understand for you. I will explain:
Your wallet is : f1b5qxhzpyg3djvqwilnksd3o3qng2xpzstn3yx2a
On the Fil+ website you can check all the storage deals made from this wallet : https://filplus.d.interplanetary.one/clients/f01920758
For example:
Looking at the top one for example gives :
lotus state get-deal 19367369 { "Proposal": { "PieceCID": { "/": "baga6ea4seaqfw77d46yu2fhkrio2ekwxecss7th6eyfxkg2k7xhs6qeftdscikq" }, "PieceSize": 34359738368, "VerifiedDeal": true, "Client": "f01920758", "Provider": "f01981603", "Label": "uAXASIJhx2uuqjTbc2vcTxGG8ee2oDQ2yxv9k0VEKSPEAWLIg", "StartEpoch": 2447959, "EndEpoch": 3989581, "StoragePricePerEpoch": "0", "ProviderCollateral": "8715519941773544", "ClientCollateral": "0" }, "State": { "SectorStartEpoch": 2439133, "LastUpdatedEpoch": 2499209, "SlashEpoch": -1, "VerifiedClaim": 1883427 } }
So it is all one chain. Now doing the retrieval :
lotus client retrieve --provider f01981603 uAXASIJhx2uuqjTbc2vcTxGG8ee2oDQ2yxv9k0VEKSPEAWLIg test.car ERROR: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid bafybeieyohnoxkung3onv5ytyrq3y6pnvagq3mwg75sncuikjdyqawfsea: getting pieces containing block bafybeieyohnoxkung3onv5ytyrq3y6pnvagq3mwg75sncuikjdyqawfsea: failed to lookup index for mh 12209871daebaa8d36dcdaf713c461bc79eda80d0db2c6ff64d1510a48f10058b220, err: datastore: key not found
As in simple English ... this data cannot be retrieved
Btw..... i appreciate you say that you work hard for it but 2 miners share the same IP address.
f01964253 -> {12D3KooWRkHMyCsagMwmJTsBh2YDndPkYbN3jC7S1ksqMg4pGxib: [/ip4/120.133.132.150/tcp/12910]} ERROR: failed to parse multiaddr "f01964253": must begin with /
root@proposals:~# lotus net connect f01964215 f01964215 -> {12D3KooWMm54HHNegB6gb7Eqxbc1sA3HSRSraxLQ8SD1J66Q2aMy: [/ip4/14.17.106.229/tcp/12891]} ERROR: failed to parse multiaddr "f01964215": must begin with /
root@proposals:~# lotus net connect f01981603 f01981603 -> {12D3KooWMtFop2smonDVRnHtx7nf6oQFiRm7uynS546P3E7xFG2m: [/ip4/39.109.125.191/tcp/10245]} ERROR: failed to parse multiaddr "f01981603": must begin with /
root@proposals:~# lotus net connect f01919535 f01919535 -> {12D3KooWCkn1gnXsGDYC2kyBCmWwyRSFsK2LgivdUqcfGwuYr6Rj: [/ip4/39.109.125.191/tcp/13904]} ERROR: failed to parse multiaddr "f01919535": must begin with /
Can you tell us who they belong to? ( f01981603 / f01919535 ) because again, this is against the rules
This client contacted me on slack to review the LDN, but I noticed that there are still some disputes that have not been resolved, and I will continue to pay attention to the LDN situation of this client.
After communicating with SP companies about data retrieval issues. Since the SP encapsulation process uses a distributed miner, the seal miner and wdpost miner exist on different servers. And dagstorge exists on the seal miner. The explanation they gave me was that all dagstorge will be imported to wdpost miner after all packaging tasks are completed. The data will then be retrievable.
The IPs of f01981603 / f01919535 are the same. Our agreement is that the SP storage company will help us store data, but there is no agreement that he can only provide a unique Node, so we will not explain this further. If you have an idea, please provide suggestions in the community, and write the rules you want into the node consensus through legal channels, so that everyone can avoid disputes.
I have posted this issue in the T&T WG for guidance.
checker:manualTrigger
ShaoxingQiaonao Enterprise Management Consulting Co., Ltd.
f1b5qxhzpyg3djvqwilnksd3o3qng2xpzstn3yx2a
2
Alex118012
kernelogic1
NDLABS-OFFICE1
newwebgroup1
PluskitOfficial1
psh0691
The below table shows the distribution of storage providers that have stored data for this client.
If this is the first time a provider takes verified deal, it will be marked as new
.
For most of the datacap application, below restrictions should apply.
⚠️ f01964215 has sealed 31.63% of total datacap.
⚠️ 83.53% of total deal sealed by f01964215 are duplicate data.
⚠️ f01981603 has sealed 42.24% of total datacap.
⚠️ 87.67% of total deal sealed by f01981603 are duplicate data.
Provider | Location | Total Deals Sealed | Percentage | Unique Data | Duplicate Deals |
---|---|---|---|---|---|
f01964253new |
Guangzhou, Guangdong, CNChina Unicom Guangdong IP network |
14.59 TiB | 2.43% | 14.59 TiB | 0.00% |
f01964215 | Shenzhen, Guangdong, CNCHINANET-BACKBONE |
189.78 TiB | 31.63% | 31.25 TiB | 83.53% |
f01981603 | Hong Kong, Central and Western, HKHONG KONG BRIDGE INFO-TECH LIMITED |
253.44 TiB | 42.24% | 31.25 TiB | 87.67% |
f01919535 | Hong Kong, Central and Western, HKHONG KONG BRIDGE INFO-TECH LIMITED |
120.00 TiB | 20.00% | 107.63 TiB | 10.31% |
f01986236 | Hong Kong, Central and Western, HKHONG KONG BRIDGE INFO-TECH LIMITED |
22.19 TiB | 3.70% | 22.19 TiB | 0.00% |
The below table shows how each many unique data are replicated across storage providers.
⚠️ 100.00% of deals are for data replicated across less than 4 storage providers.
Unique Data Size | Total Deals Made | Number of Providers | Deal Percentage |
---|---|---|---|
133.34 TiB | 284.47 TiB | 1 | 47.41% |
36.78 TiB | 315.53 TiB | 2 | 52.59% |
The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.
However, this could be possible if all below clients use same software to prepare for the exact same dataset or they belong to a series of LDN applications for the same dataset.
⚠️ CID sharing has been observed.
Other Client | Application | Total Deals Affected | Unique CIDs | Approvers |
---|---|---|---|---|
f3soxnqjhnqmerbnpzanizjlqfkjuxsshvhvwntgh cfwtoui3j7vmfah6zgv6fmh3ce2yo2bfacgt7sfud liya |
Unknown | 64.00 GiB | 1 | Unknown |
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
From the node connectivity check displayed by the check bot, except for the newly added node, other nodes are connected normally. But the regions are all together.
f3soxnqjhnqmerbnpzanizjlqfkjuxsshvhvwntgh cfwtoui3j7vmfah6zgv6fmh3ce2yo2bfacgt7sfud liya -- I have not checked the client at this address, please check with other notaries
@NDLABS-OFFICE
@cryptowhizzard About the IPs of f01981603 / f01919535 are the same. In order to resume this project as soon as possible, I decided to solve this problem, I proposed a plan to contact SP to carry out the whole migration of f01981603, this will take some time because of the need to move multiple rack mounted servers and the need to avoid the Proof-Of-Space-Time period, but I promise to carry out.
Until there is a clear list with SP's their business names and region I am not supportive of this LDN. Distributing only to Asian SP's is not the right way to make use of datacap in my opinion.
If you have the bandwidth to go to SG, HK or JP, you can also send those deals to USA and EU.
SG, HK or JP, USA or EU
@herrehesse Ok, it is exactly my plan for the next round
@zhongdonglin will you let us know when you are ready?
Next round plan:
@cryptowhizzard We are ready for this , and now we looking for signatures for the next round.
Hi @zhongdonglin
The SP's above have been involved in CID sharing. I can't sign on those unfortunately.
@cryptowhizzard From the information I already have, none of the MinerIDs I mentioned above has the problem of CID sharing. If you insist on your claim, you should provide corresponding evidence. Is this reasonable?
@zhongdonglin
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/356
cntrl -:> F + f01876488
Have a nice day.
@cryptowhizzard If so, shouldn't you persuade me to revise my plans? This SP has not yet participated in my project, and it is completely avoidable, so all this can be adjusted.
@cryptowhizzard If so, shouldn't you persuade me to revise my plans? This SP has not yet participated in my project, and it is completely avoidable, so all this can be adjusted.
Hi,
Sure, let me know if you have another SP and i will check for you. Np.
From your answer i understood that you were a bot aggravated b/c you asked for proof etc. Reason i went to close the convo to avoid further struggles. Sorry if I misunderstood.
@cryptowhizzard how about the SP-f01777785 in SG?
Hi @zhongdonglin
SP's f01777785 flag came from this LDN. , however that is cleared now since 48 hours. I will unflag it , the SP is ok!
Moving forward with this and making the plan : You will store with f01777785 and -> Resolve duplicate data problem of f01964215 Reduce the overall proportion of f01964215 and f01981603
If so, it looks ok to me and I can propose if you please can fill out https://form.jotform.com/230337462961356 with your name etc.
@cryptowhizzard Already submited the form, pls check it.
Your Datacap Allocation Request has been approved by the Notary
bafy2bzacednvmkly3w234dfbuura5oknciuwbahdkfadej6dp6dtbfgppvhp4
Address
f1b5qxhzpyg3djvqwilnksd3o3qng2xpzstn3yx2a
Datacap Allocated
640.00TiB
Signer Address
f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa
Id
9a357e58-feb4-43f2-8b71-269dc128ad40
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacednvmkly3w234dfbuura5oknciuwbahdkfadej6dp6dtbfgppvhp4
f01858410
f1b5qxhzpyg3djvqwilnksd3o3qng2xpzstn3yx2a
640TiB
cb6a70e0-414a-4fa1-92a1-d8f81a8c577a
f01858410
f1b5qxhzpyg3djvqwilnksd3o3qng2xpzstn3yx2a
cryptowhizzard & not found
800% of weekly dc amount requested
640TiB
600TiB
1.41PiB
Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
---|---|---|---|---|
19943 | 5 | 640TiB | 42 | 0B |
⚠️ 2 storage providers sealed more than 30% of total datacap - f01964215: 31.63%, f01981603: 42.24%
⚠️ 2 storage providers sealed too much duplicate data - f01964215: 83.53%, f01981603: 87.67%
⚠️ 100.00% of deals are for data replicated across less than 4 storage providers.
⚠️ CID sharing has been observed. (Top 3)
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Click here to view the full report.
Maybe you need to confirm whether the data download is abnormal. In addition, the dispersion. Become better and better.
@Joss-Hua It's Ok now i test.
Hi @jamerduhgamer, we have answered your questions in detail. Is it convenient for you to sign now?
checker:manualTrigger
⚠️ 1 storage providers sealed more than 30% of total datacap - f01981603: 38.24%
⚠️ 2 storage providers sealed too much duplicate data - f01964215: 83.53%, f01981603: 87.67%
⚠️ 100.00% of deals are for data replicated across less than 4 storage providers.
⚠️ CID sharing has been observed. (Top 3)
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Click here to view the full report.
hope the dispersion can be better
Your Datacap Allocation Request has been proposed by the Notary
bafy2bzacecqkal3euxdxuir42lkebswmqvbldpm3454lvhka6lkfvdl2dxed2
Address
f1b5qxhzpyg3djvqwilnksd3o3qng2xpzstn3yx2a
Datacap Allocated
640.00TiB
Signer Address
f1tfg54zzscugttejv336vivknmsnzzmyudp3t7wi
Id
cb6a70e0-414a-4fa1-92a1-d8f81a8c577a
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecqkal3euxdxuir42lkebswmqvbldpm3454lvhka6lkfvdl2dxed2
Checked the CID Checker report, there is no CID sharing, and the alert item Client has a response, which is within the acceptable range. Hopefully, in the next round, the SP positions will be more spread out and the data distribution will be more reasonable. Willing to support this round, will keep watching and monitoring.
SP Support Retrieval lotus client retrieve --provider f01919535 bafybeidjflun45gcktwyxavzocrgvvajedt76m4eazxamgav4qm7v6wjrq LDN-975-2
Your Datacap Allocation Request has been approved by the Notary
bafy2bzaced77hbpifooyaixhr7c7ygbnz5xuuglmlbgjj56gw7cmymbzp3plm
Address
f1b5qxhzpyg3djvqwilnksd3o3qng2xpzstn3yx2a
Datacap Allocated
640.00TiB
Signer Address
f1e77zuityhvvw6u2t6tb5qlnsegy2s67qs4lbbbq
Id
cb6a70e0-414a-4fa1-92a1-d8f81a8c577a
You can check the status of the message here: https://filfox.info/en/message/bafy2bzaced77hbpifooyaixhr7c7ygbnz5xuuglmlbgjj56gw7cmymbzp3plm
This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.
checker:manualTrigger
⚠️ 2 storage providers sealed too much duplicate data - f01964215: 53.71%, f01981603: 87.67%
⚠️ 71.67% of deals are for data replicated across less than 4 storage providers.
⚠️ CID sharing has been observed. (Top 3)
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.
Large Dataset Notary Application
To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.
Core Information
Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.
Project details
Share a brief history of your project and organization.
What is the primary source of funding for this project?
What other projects/ecosystem stakeholders is this project associated with?
Use-case details
Describe the data being stored onto Filecoin
Where was the data in this dataset sourced from?
Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this.
Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).
What is the expected retrieval frequency for this data?
For how long do you plan to keep this dataset stored on Filecoin?
DataCap allocation plan
In which geographies (countries, regions) do you plan on making storage deals?
How will you be distributing your data to storage providers? Is there an offline data transfer process?
How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.
How will you be distributing deals across storage providers?
Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?