Closed xinaxu closed 11 months ago
Best practice for storing large datasets includes ideally, storing it in 3 or more regions, with 4 or more storage provider operators or owners. You should list Miner ID, Business Entity, Location of sps in one more region. Could you send an email to filplus-app-review@fil.org with your official domain in order to confirm your identity? Email name should includes the issue id #2087 .
@Sunnyiscoming
You should list Miner ID, Business Entity, Location of sps in one more region.
See below for the latest miner list
f01989866 - Weimin, Xi'an, CN
f09848 - bigbear, CA, US
f01390323 - Lianxing storage, Hangzhou, CN
f081990 - Weimin Pan, HK
f01967501, f01717477, f02214491 - Acrontech, PA, US
f02228866 - Feige IT, Tokyo, JP
f01315130 - Ouruan IT, Chengdu, CN
Could you send an email to
Done
Have you prepared enough token for sector pledge
I have not, but my storage providers have enough token to start pledging
Are you a data preparer? What is your previous experience as a data-preparer? List previous applications and client IDs
Yes, I am very familiar with data preparation. See below list of all previous LDN applications https://github.com/filecoin-project/filecoin-plus-large-datasets/issues?q=is%3Aissue+is%3Aopen+tech+greedy
How will the data be prepared? Please include tooling used and technical details
It will be prepared using Singularity V2 - Technical details
Has this dataset been stored on Filecoin before? If so, why are you choosing to store it again?
We are THE FIRST data preparers that works on this dataset.
What's your website?
We don't have website, most of our work are done via open source platform github.com or internal slack channel. You can find our online presence in multiple places
We are a Filecoin Plus Notary https://github.com/filecoin-project/notary-governance/issues/664
I am featured in media post for the development work https://mirror.xyz/joinradius.eth/vwmrtaEcas5Ei3Ff6kNor6FB2i9zTvp0SMRuxOv6l_A
The storage software we developed are widely adopted by Filecoin clients https://github.com/tech-greedy
Total DataCap requested
15PiB
Expected weekly DataCap usage rate
1PiB
Client address
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
f02049625
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
512TiB
1adc4140-6591-4230-b667-ce03eec6dc05
Your Datacap Allocation Request has been proposed by the Notary
bafy2bzacedrp5a576fvyr6u5cptb5yls6omnemcnqn5s56wlew7rno6basyiw
Address
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
Datacap Allocated
512.00TiB
Signer Address
f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa
Id
1adc4140-6591-4230-b667-ce03eec6dc05
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedrp5a576fvyr6u5cptb5yls6omnemcnqn5s56wlew7rno6basyiw
Hello @xinaxu Hello per the new guidelines https://github.com/filecoin-project/notary-governance/issues/922 for Open Dataset applicants, we are asking all applicants to complete the following Fil+ registration form to identify yourself as the applicant and also please add the contact information of the SP entities you are working with to store copies of the data.
We see that you have provided this information above in the comments so that notaries now have information needed to complete their due diligence toward your project meeting Fil+ distributed storage guidelines. Thank you, this application may proceed.
Your Datacap Allocation Request has been approved by the Notary
bafy2bzaceb3bw5kecponi2evieeoiyu4i5xku75fcol6e44x67q6driapgrpa
Address
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
Datacap Allocated
512.00TiB
Signer Address
f1xzff5xup63o5sygr2swp4zvcajg54lotliimdty
Id
1adc4140-6591-4230-b667-ce03eec6dc05
You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceb3bw5kecponi2evieeoiyu4i5xku75fcol6e44x67q6driapgrpa
This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.
keep alive
f02049625
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
512TiB
0b3f144a-22ee-4016-a12f-ee9982603333
f02049625
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
100% weekly > 0.5PiB, requesting 0.5PiB
512TiB
512TiB
14.5PiB
Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
---|---|---|---|---|
5953 | 4 | 512TiB | 55.39 | 60.92TiB |
checker:manualTrigger
✔️ Storage provider distribution looks healthy.
⚠️ 99.95% of deals are for data replicated across less than 3 storage providers.
⚠️ CID sharing has been observed. (Top 3)
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.
Your Datacap Allocation Request has been proposed by the Notary
bafy2bzacec756mno5rnvxefjaognnmued2ukmzzpe7gjtak5cpwu3335fztq4
Address
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
Datacap Allocated
512.00TiB
Signer Address
f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa
Id
0b3f144a-22ee-4016-a12f-ee9982603333
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec756mno5rnvxefjaognnmued2ukmzzpe7gjtak5cpwu3335fztq4
Reviewed Sunnyiscoming's questions above and Xinaxu's answers to them were sufficient responses. Dataset is also being stored in SPs in 3 different countries. I see no outstanding issues with supporting this LDN.
Your Datacap Allocation Request has been approved by the Notary
bafy2bzacec4uzwkljy43zbgq3ytn3ewvxd67l4kat3knm3vqwwwwfbovjlwuo
Address
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
Datacap Allocated
512.00TiB
Signer Address
f1kqdiokoeubyse4qpihf7yrpl7czx4qgupx3eyzi
Id
0b3f144a-22ee-4016-a12f-ee9982603333
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec4uzwkljy43zbgq3ytn3ewvxd67l4kat3knm3vqwwwwfbovjlwuo
This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.
keep open
f02049625
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
1PiB
cc71d766-f4fb-49bd-9b56-6d31f9976547
f02049625
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
200% weekly > 1PiB, requesting 1PiB
1PiB
465661.3YiB
465661.3YiB
Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
---|---|---|---|---|
24906 | 7 | 512TiB | 29.39 | 2.07TiB |
checker:manualTrigger
✔️ Storage provider distribution looks healthy.
⚠️ 99.93% of deals are for data replicated across less than 4 storage providers.
⚠️ CID sharing has been observed. (Top 3)
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.
Can you explain the CID sharing problem and also Deal Data Replication abnormality before next tranche?
checker:manualTrigger f1mshuxqsmbaxtov7akpua76orqthwseotvwfu5ky f1l2cc5vuw5moppwsjd3b7cjtwa2exowqo36esklq f13vtwldyycj32sxhenrd7gmwj72hhatvuoydjxii
f1mshuxqsmbaxtov7akpua76orqthwseotvwfu5ky - NationalOceanic and Atmospheric Administration (NOAA)
f1l2cc5vuw5moppwsjd3b7cjtwa2exowqo36esklq - NationalOceanic and Atmospheric Administration (NOAA)
f13vtwldyycj32sxhenrd7gmwj72hhatvuoydjxii - NationalOceanic and Atmospheric Administration (NOAA)
⚠️ 1 storage providers sealed too much duplicate data - f02240216: 29.65%
✔️ Data replication looks healthy.
⚠️ CID sharing has been observed. (Top 3)
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.
@spaceT9 The reason for cid sharing is, those are the same dataset prepared by my org. The low replication is because this LDN is for replications to more miners, you can check the aggregated stats above to see the overal replication.
Fair explanation. I will support this round.
Your Datacap Allocation Request has been proposed by the Notary
bafy2bzacecdasznzxxeyb6jbbfw26555n4zv6dxpaffatv2crf36uofe3l5he
Address
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
Datacap Allocated
1.00PiB
Signer Address
f1a2lia2cwwekeubwo4nppt4v4vebxs2frozarz3q
Id
cc71d766-f4fb-49bd-9b56-6d31f9976547
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecdasznzxxeyb6jbbfw26555n4zv6dxpaffatv2crf36uofe3l5he
Client has explained to me the duplicate data was caused by Singularity failed deal status tracking and client has taken necessary steps to correct it. As the author of the singularity deal tracking module I can confirm that can happen and it was a technical accident.
Otherwise the reports looks OK.
Your Datacap Allocation Request has been approved by the Notary
bafy2bzacebf7rr2j3q5pzvxwzestdwgh6lupm5j7cka7q6a3jk5dyv5mee4qa
Address
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
Datacap Allocated
1.00PiB
Signer Address
f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa
Id
cc71d766-f4fb-49bd-9b56-6d31f9976547
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebf7rr2j3q5pzvxwzestdwgh6lupm5j7cka7q6a3jk5dyv5mee4qa
checker:manualTrigger
✔️ Storage provider distribution looks healthy.
⚠️ 99.36% of deals are for data replicated across less than 4 storage providers.
⚠️ CID sharing has been observed. (Top 3)
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.
I'm working with f09848 and f01390323 to turn on free retrieval
f02049625
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
2PiB
6ec5118d-9047-4dd0-bd18-364fd73ea908
f02049625
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
400% weekly > 2PiB, requesting 2PiB
2PiB
931322574615478927360.0YiB
931322574615478927360.0YiB
Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
---|---|---|---|---|
58377 | 9 | 1PiB | 42.43 | 2.40TiB |
Your Datacap Allocation Request has been proposed by the Notary
bafy2bzaceda52bg4w7iezlmedjho63hgxdsf43su3x6htwvbt3vigndtm7kfs
Address
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
Datacap Allocated
2.00PiB
Signer Address
f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa
Id
6ec5118d-9047-4dd0-bd18-364fd73ea908
You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceda52bg4w7iezlmedjho63hgxdsf43su3x6htwvbt3vigndtm7kfs
checker:manualTrigger
⚠️ 1 storage providers sealed more than 30% of total datacap - f02361686: 36.99%
⚠️ 94.05% of deals are for data replicated across less than 4 storage providers.
⚠️ CID sharing has been observed. (Top 3)
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.
Hi @xinaxu, will you be onboarding more SPs to the project?
checker:manualTrigger f1mshuxqsmbaxtov7akpua76orqthwseotvwfu5ky f1l2cc5vuw5moppwsjd3b7cjtwa2exowqo36esklq f13vtwldyycj32sxhenrd7gmwj72hhatvuoydjxii
f1mshuxqsmbaxtov7akpua76orqthwseotvwfu5ky - NationalOceanic and Atmospheric Administration (NOAA)
f1l2cc5vuw5moppwsjd3b7cjtwa2exowqo36esklq - NationalOceanic and Atmospheric Administration (NOAA)
f13vtwldyycj32sxhenrd7gmwj72hhatvuoydjxii - NationalOceanic and Atmospheric Administration (NOAA)
⚠️ 1 storage providers sealed too much duplicate data - f02240216: 25.07%
✔️ Data replication looks healthy.
⚠️ CID sharing has been observed. (Top 3)
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.
Your Datacap Allocation Request has been approved by the Notary
bafy2bzacecfnpt2qe6nflahj73icabhyyjm7kco7gc2twb66socrjn7byt5b4
Address
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
Datacap Allocated
2.00PiB
Signer Address
f1kqdiokoeubyse4qpihf7yrpl7czx4qgupx3eyzi
Id
6ec5118d-9047-4dd0-bd18-364fd73ea908
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecfnpt2qe6nflahj73icabhyyjm7kco7gc2twb66socrjn7byt5b4
f02049625
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
2PiB
fb0301e5-3832-4b6f-adf3-90c4a9601ebb
f02049625
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
400% weekly > 2PiB, requesting 2PiB
2PiB
1.862645149230957e+37YiB
1.862645149230957e+37YiB
Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
---|---|---|---|---|
61843 | 9 | 2PiB | 40.21 | 1.90PiB |
checker:manualTrigger
⚠️ 1 storage providers sealed more than 30% of total datacap - f02361686: 34.28%
⚠️ 93.61% of deals are for data replicated across less than 4 storage providers.
⚠️ CID sharing has been observed. (Top 3)
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.
Data Owner Name
National Oceanic and Atmospheric Administration (NOAA)
What is your role related to the dataset
Data Preparer
Data Owner Country/Region
United States
Data Owner Industry
Environment
Website
https://registry.opendata.aws/noaa-gefs/
Social Media
Total amount of DataCap being requested
15PiB
Expected size of single dataset (one copy)
3 PiB
Number of replicas to store
10
Weekly allocation of DataCap requested
1PiB
On-chain address for first allocation
f1fjegj5ihpjn5ie75np6vagpwbvx3je4ni2ve4iq
Data Type of Application
Public, Open Dataset (Research/Non-Profit)
Custom multisig
Identifier
No response
Share a brief history of your project and organization
Is this project associated with other projects/ecosystem stakeholders?
Yes
If answered yes, what are the other projects/ecosystem stakeholders
Describe the data being stored onto Filecoin
Where was the data currently stored in this dataset sourced from
AWS Cloud
If you answered "Other" in the previous question, enter the details here
No response
How do you plan to prepare the dataset
singularity
If you answered "other/custom tool" in the previous question, enter the details here
No response
Please share a sample of the data
Confirm that this is a public dataset that can be retrieved by anyone on the Network
If you chose not to confirm, what was the reason
No response
What is the expected retrieval frequency for this data
Yearly
For how long do you plan to keep this dataset stored on Filecoin
Permanently
In which geographies do you plan on making storage deals
Greater China, Asia other than Greater China, North America, Europe
How will you be distributing your data to storage providers
Cloud storage (i.e. S3), HTTP or FTP server
How do you plan to choose storage providers
Slack, Big Data Exchange, Partners
If you answered "Others" in the previous question, what is the tool or platform you plan to use
No response
If you already have a list of storage providers to work with, fill out their names and provider IDs below
How do you plan to make deals to your storage providers
Singularity
If you answered "Others/custom tool" in the previous question, enter the details here
No response
Can you confirm that you will follow the Fil+ guideline
Yes