Closed galen-mcandrew closed 1 year ago
Thanks for your request! Everything looks good. :ok_hand:
A Governance Team member will review the information provided and contact you back pretty soon.
@starling-admin Here is the new large dataset application issue, per the new LDN process.
5PiB
Expected weekly DataCap usage rate
100TiB
**Multisig created and sent to RKH f01322626
f01322626
f3w5fx6wta4ewl2iyf7xcogmzffz2fmrngpzdpduj3xmk3dwjxc6dyq36gdf3rflkkrblh5nci5xymc5hal3qq
50TiB
Your Datacap Allocation Request has been approved by the Notary
bafy2bzacebmb5eshdszn6mike6iohpbstp7yin4evmiaszriuu2ccxlpimjpo
Address
f3w5fx6wta4ewl2iyf7xcogmzffz2fmrngpzdpduj3xmk3dwjxc6dyq36gdf3rflkkrblh5nci5xymc5hal3qq
Datacap Allocated
50TiB
Signer Address
f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebmb5eshdszn6mike6iohpbstp7yin4evmiaszriuu2ccxlpimjpo
Your Datacap Allocation Request has been approved by the Notary
bafy2bzaceav3tvsfrq57kzh4ldst2uynaf6ukbtpljyvcavsw7dmqtvkcr4yy
Address
f3w5fx6wta4ewl2iyf7xcogmzffz2fmrngpzdpduj3xmk3dwjxc6dyq36gdf3rflkkrblh5nci5xymc5hal3qq
Datacap Allocated
50TiB
Signer Address
f1k6wwevxvp466ybil7y2scqlhtnrz5atjkkyvm4a
You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceav3tvsfrq57kzh4ldst2uynaf6ukbtpljyvcavsw7dmqtvkcr4yy
Your Datacap Allocation Request has been approved by the Notary
bafy2bzacecqgfrguyxw33sg5saijaj2jtbkcwzxfcmyhslznvzesvx7rtamzc
Address
f3w5fx6wta4ewl2iyf7xcogmzffz2fmrngpzdpduj3xmk3dwjxc6dyq36gdf3rflkkrblh5nci5xymc5hal3qq
Datacap Allocated
50TiB
Signer Address
f1k6wwevxvp466ybil7y2scqlhtnrz5atjkkyvm4a
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecqgfrguyxw33sg5saijaj2jtbkcwzxfcmyhslznvzesvx7rtamzc
^ this went as a proposal, looks like a dup. I'm going to remove the ready to sign label here since the allocation was already made.
@dkkapur + the Datacap notaries,
We'd like to move our next tranche of our approved data cap allocation to this wallet: f17g7h52bsi53rb263xwne573dusskit4mieqkgry
We are actively sealing deals and ready to scale.
Thanks!
According to glif, address f3w5fx6wta4ewl2iyf7xcogmzffz2fmrngpzdpduj3xmk3dwjxc6dyq36gdf3rflkkrblh5nci5xymc5hal3qq has 14 TiB remaining. With the initial allocation of 50TiB, that means the next allocation should kick off at ~12 TiB.
Checking lotus, seeing 2 pending transactions with only 1 approval, so I want to check in with the team before I make any changes to the client address.
@fabriziogianni7 @ialberquilla
Additionally flagging for notaries: @starling-admin according to a quick audit (https://filplus.d.interplanetary.one/clients/f0700600/breakdown), while you have worked with 20 storage providers, an extreme majority of your deal-making has been with a single storage provider (84%).
Per your application and the large dataset process, can you provide some more details about your deal distribution plan?
Hi Galen,
We are still testing the deal automation with Bitbot so we are progressively decentralizing. The plan so far is as follows:
Pilot 81 TiBs sealed with 1 provider (as a test) Pilot 81 TiBs sealed with 4 providers Pilot 81 TiBs sealed with 10 providers
With these pilots finished we are then going to work on a plan for 4 PiBs. The plan there is to distribute to an initial set of 10 providers and then scale to as many as possible. The fulcrum will be what type of financing SPs will need to take on to meet their pledge. We are working with PL on that mechanism.
Happy to answer any questions you have.
Best, Jonathan
On Thu, Dec 16, 2021 at 3:49 PM Galen @.***> wrote:
Additionally flagging for notaries: @starling-admin https://github.com/starling-admin according to a quick audit (https://filplus.d.interplanetary.one/clients/f0700600/breakdown), while you have worked with 20 storage providers, an extreme majority of your deal-making has been with a single storage provider (84%).
Per your application and the large dataset process, can you provide some more details about your deal distribution plan?
— Reply to this email directly, view it on GitHub https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/53#issuecomment-996281761, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP5M66UQ3V735S4HL3GCNXTURJ3HRANCNFSM5FDDUOOA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.*** com>
f01322626
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
100TiB
f01322626
f3w5fx6wta4ewl2iyf7xcogmzffz2fmrngpzdpduj3xmk3dwjxc6dyq36gdf3rflkkrblh5nci5xymc5hal3qq
dannyob & dannyob
100TiB
Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
---|---|---|---|---|
2762 | 20 | 50TiB | 84.06 | 11.9TiB |
Edited parent comment and the request comment with new client address, app front-end appears to be displaying correctly.
Your Datacap Allocation Request has been approved by the Notary
bafy2bzaceclyqpuoqqnnp6bzyanmlee53jppwti5cvggs556im25qqtb5mrj4
Address
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
Datacap Allocated
100TiB
Signer Address
f1ystxl2ootvpirpa7ebgwl7vlhwkbx2r4zjxwe5i
You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceclyqpuoqqnnp6bzyanmlee53jppwti5cvggs556im25qqtb5mrj4
Hi @MegTei,
It seems the network still is not aware of f17g7h52bsi53rb263xwne573dusskit4mieqkgry
being a verified client. It is unable to send verified deals and https://verify.glif.io/ reflects this state as well.
Any ideas what could be the issue?
@Ray-PiKNiK - we need one more notary to sign before this gets allocated. 2 signatures from participating notaries are required for each allocation to go through.
Your Datacap Allocation Request has been approved by the Notary
bafy2bzacede44lqrknmbg2izdisx4qesj5v4l4kojbuqdhiye6vzmundwr7a6
Address
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
Datacap Allocated
100TiB
Signer Address
f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacede44lqrknmbg2izdisx4qesj5v4l4kojbuqdhiye6vzmundwr7a6
Dear Datacap Notaries,
We are getting ready to ramp up deal distribution and would like to start the data cap tranche allocation process now in order to reduce the turnaround time needed moving forward.
Thanks!
Dear Datacap Notaries,
We are all excited to start scaling our implementation with the USC Shoah Foundation dataset. This will involve 2 key changes:
We are working now to spread the data to 4 SP (to meet the Datacap base decentralization requirement) and identified a total of 10 SPs beyond PikNik that we want to store data with. We are in the process of sending test data sets to each and are very motivated to get the data distributed as far and wide as possible.
To help us keep pace, I am deputizing @Ray-PiKNiK to handle our interactions with you going forward. This includes troubleshooting these DataCap issues and coming to you with requests to manually release FIL+ allocations
Thanks again for understanding!
HI @Ray-PiKNiK did you need anything more from me at this stage?
Hi @MegTei, we plan to start distributing the deals next week. The aggregate sealing rate of the storage providers will be at least 25 TiBs/day. With the remaining data cap available (50T) on our wallet, we would run out in 2 days.
While the allocation bot will get trigger at <25% available, we would like to preemptively request the next allocation of data cap now in preparation for next week if possible.
Thank you!
f01322626
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
200TiB
f01322626
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
flyworker & MegTei
200% of weekly dc amount requested
200TiB
100TiB
4.90PiB
Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
---|---|---|---|---|
2943 | 16 | 100TiB | 24.82 | 24.69TiB |
Your Datacap Allocation Request has been proposed by the Notary
bafy2bzacedafmdb6knk72czet55h7d6uwq3c2k6cda55wkppsp4wsrrkpmhyo
Address
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
Datacap Allocated
200.00TiB
Signer Address
f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedafmdb6knk72czet55h7d6uwq3c2k6cda55wkppsp4wsrrkpmhyo
Your Datacap Allocation Request has been approved by the Notary
bafy2bzacednk4zf5l4gu4gwz3q76vsr5wyefusrbxhypwtu6q7uaqgbcbzacs
Address
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
Datacap Allocated
200.00TiB
Signer Address
f1ystxl2ootvpirpa7ebgwl7vlhwkbx2r4zjxwe5i
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacednk4zf5l4gu4gwz3q76vsr5wyefusrbxhypwtu6q7uaqgbcbzacs
f01322626
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
400TiB
f01322626
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
MegTei & flyworker
400% of weekly dc amount requested
400TiB
100TiB
4.90PiB
Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
---|---|---|---|---|
8561 | 19 | 200TiB | 48.66 | 48.88TiB |
Your Datacap Allocation Request has been proposed by the Notary
bafy2bzaceb5kexkgrjffzo4h4nrccv5ziqhafajikg5rxiul6eygwatv2dny4
Address
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
Datacap Allocated
400.00TiB
Signer Address
f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa
You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceb5kexkgrjffzo4h4nrccv5ziqhafajikg5rxiul6eygwatv2dny4
Your Datacap Allocation Request has been approved by the Notary
bafy2bzacedkbtept24dczb7p2jpcsrwbvpdwvb3fy5vd5aa6dvpjl45arrsoe
Address
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
Datacap Allocated
400.00TiB
Signer Address
f1k6wwevxvp466ybil7y2scqlhtnrz5atjkkyvm4a
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedkbtept24dczb7p2jpcsrwbvpdwvb3fy5vd5aa6dvpjl45arrsoe
@starling-admin Could you provide your SP node and rootcid ? We need to retrieve your public dataset.
Hi @Yvette516, unfortunately the majority of dataset will be unable to retrieved as mentioned in datacap request that "some of the data has been requested to be private for a period of time based on requests from the interviewees of the content." The data will be made public after enough time has passed depending on USC SFI's decision.
You can see some samples of the files here - https://vhaonline.usc.edu/ Some of the testimonies here - https://www.youtube.com/playlist?list=PLWIFgIFN2QqiDdkA-MXpsvZOSvTYkEGsL.
@jamerduhgamer @dkkapur Before you said that the data you provided could be publicly accessible, but now it can't. You violated the rules of the whole community.
@jamerduhgamer @dkkapur Before you said that the data you provided could be publicly accessible, but now it can't. You violated the rules of the whole community.
I do not see where it was claimed that the data set would be public, here is the actual question and response:
Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).
A lot of the data is already publicly available for view via the Visual History Archive (https://vhaonline.usc.edu/). Some of the data has been requested to be private for a period of time based on requests from the interviewees of the content.
Having worked directly with the Shoah Foundation and this project for the last year, it was clearly articulated that this data set is not public and needs to be kept private to protect the privacy and lives of those who survived the genocide attempt and recorded their testimonies for posterity.
I think there is a miscommunication going on @Yvette516
PiKNiK send this data to different SP’s across the globe and you can retrieve that data. What is being referred to is the dataset itself. A small part of it is not send to SP’s for public use to store and have retrievable.
From the deals we (Dcent) received from PikNik you are welcome to retrieve them!
f01858410
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
800TiB
f01858410
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
dannyob & @cryptowhizzard
800% of weekly dc amount requested
800TiB
500TiB
4.51PiB
Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
---|---|---|---|---|
20485 | 21 | 400TiB | 62.49 | 86.73TiB |
Your Datacap Allocation Request has been proposed by the Notary
bafy2bzaceajdkmdf3vhw2t474mqs535rmemcammen7pumh7rxp47csi7bh3uy
Address
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
Datacap Allocated
800.00TiB
Signer Address
f1ystxl2ootvpirpa7ebgwl7vlhwkbx2r4zjxwe5i
You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceajdkmdf3vhw2t474mqs535rmemcammen7pumh7rxp47csi7bh3uy
Your Datacap Allocation Request has been approved by the Notary
bafy2bzacec5fgcjt2d6haovaxt2ga7plbv2o2nztf5poyrlldugobzlbkl3og
Address
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
Datacap Allocated
800.00TiB
Signer Address
f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec5fgcjt2d6haovaxt2ga7plbv2o2nztf5poyrlldugobzlbkl3og
As @Sunnyiscoming pointed out in https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/420#issuecomment-1173411027, a single SP was allocating with an extremely high DataCap which is not a good way to backup data and violating Filecoin rules. While @Ray-PikNik did make an explanation but it was a rogue one. Yeah "in the long term" each SP will have an even share, but 16 days since the explanation, nothing has changed, f01833311 still holds over 62% of total DataCap. So if "long term" means a decade or even a century, that absolutely ridiculous. Community rules for everyone will make no sense. Besides, I requested to have rootCIDs but was refused since "some of the data has been requested to be private for a period of time based on requests from the interviewees of the content.". Again they need "a period of time", a rogue content that make no sense at all. We need a roadmap, a precise one. Leaving this behind, "some of the data is private", so we do have some rest of the data that are publicly retrievable aren't they? Why can't you provide rootCIDs for supervision? To sum up, I propose to re-evaluate this application and concerning issue #420 until a detailed, precise and reasonable explanation is given.
f01858410
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
800TiB
f01858410
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
kernelogic & megtei
800% of weekly dc amount requested
800TiB
800TiB
4.21PiB
Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
---|---|---|---|---|
20208 | 6 | 800TiB | 37.95 | 199.82TiB |
Your Datacap Allocation Request has been proposed by the Notary
bafy2bzacebhkixh3tp4zqmhwdx5hog2helm3fmekutwo2u22jen6uh5jkfdhe
Address
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
Datacap Allocated
800.00TiB
Signer Address
f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebhkixh3tp4zqmhwdx5hog2helm3fmekutwo2u22jen6uh5jkfdhe
Your Datacap Allocation Request has been approved by the Notary
bafy2bzacea3kfmzkfvcwhhm7s2ljtwyv7jcadsjbtd5nrtr3hgwzlpu2wemgo
Address
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
Datacap Allocated
800.00TiB
Signer Address
f1k6wwevxvp466ybil7y2scqlhtnrz5atjkkyvm4a
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea3kfmzkfvcwhhm7s2ljtwyv7jcadsjbtd5nrtr3hgwzlpu2wemgo
As @Sunnyiscoming pointed out in #420 (comment), a single SP was allocating with an extremely high DataCap which is not a good way to backup data and violating Filecoin rules. While @Ray-PiKNiK did make an explanation but it was a rogue one. Yeah "in the long term" each SP will have an even share, but 16 days since the explanation, nothing has changed, f01833311 still holds over 62% of total DataCap. So if "long term" means a decade or even a century, that absolutely ridiculous. Community rules for everyone will make no sense. Besides, I requested to have rootCIDs but was refused since "some of the data has been requested to be private for a period of time based on requests from the interviewees of the content.". Again they need "a period of time", a rogue content that make no sense at all. We need a roadmap, a precise one. Leaving this behind, "some of the data is private", so we do have some rest of the data that are publicly retrievable aren't they? Why can't you provide rootCIDs for supervision? To sum up, I propose to re-evaluate this application and concerning issue #420 until a detailed, precise and reasonable explanation is given.
@starling-admin just want remind you it's been over a month now and I'm still waiting for your answers.
f01833311 still holds 46% of the total allocation. This is extremely high beyond what we generally consider to be a common range. With such a high percentage (which was even higher before), I don't understand why you didn't disclose more detail about this SP. This's not how transparency should be. Also @galen-mcandrew the latest two rounds of multisig were done by @kernelogic. This is a clear violation of community rules.
Hello @Yvette516, we have stopped sending deals to f01833311 since they are slightly higher than the 30% requirement. You can check on Filfox here - https://filfox.info/en/address/f01833311 and this SP has not grown in power since July 31st, 7/31.
It seems like a bug that Kernelogic was able to approve two rounds of multisig in a row. We should implement a process that does not allow this to happen.
f01858410
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
800TiB
a8b70877-afe7-49e6-87c0-17ff7244b5a5
f01858410
f17g7h52bsi53rb263xwne573dusskit4mieqkgry
dannyob & kernelogic
800% of weekly dc amount requested
800TiB
1.56PiB
3.43PiB
Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
---|---|---|---|---|
50828 | 9 | 800TiB | 22.07 | 199.72TiB |
Large Dataset Notary Application
To apply for a DataCap allocation for your dataset, please fill out the following information.
Core Information
Please respond to the questions below in pargraph form, replacing the text saying "Please answer here". Include as much detail as you can in your answer!
Project details
Share a brief history of your project and organization.
What is the primary source of funding for this project?
What other projects/ecosystem stakeholders is this project associated with?
Use-case details
Describe the data being stored onto Filecoin
Where was the data in this dataset sourced from?
Can you share a sample of what is in the dataset? A link to a file, an image, a table, etc., are good examples of this.
Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).
What is the expected retrieval frequency for this data?
For how long do you plan to keep this dataset stored on Filecoin? Will this be a permanent archival or a one-time storage deal?
DataCap allocation plan
In which geographies do you plan on making storage deals?
What is your expected data onboarding rate? How many deals can you make in a day, in a week? How much DataCap do you plan on using per day, per week?
How will you be distributing your data to miners? Is there an offline data transfer process?
How do you plan on choosing the miners with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.
How will you be distributing data and DataCap across miners storing data?