search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

Victor Chang Cardiac Research Institute #425

Closed DSS-AL closed 1 year ago

DSS-AL commented 2 years ago

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Share a brief history of your project and organization.

The Victor Chang Cardiac Research Institute (VCCRI) is renowned for the quality of its [scientific discoveries](https://www.victorchang.edu.au/heart-research/major-discoveries) and is dedicated to finding cures for cardiovascular disease through world-class and cutting-edge [medical research](https://www.victorchang.edu.au/heart-research).
DSS have worked with VCCRI to develop a PoC to demonstrate the operational and economic benefit of the Filecoin Network and subsequently make this application on their behalf to solve a long-term data storage requirement resulting from their research.
VCCRI are seeking to store five copies of a 1 PiB dataset as an archive on the Filecoin Network.
DSS is a leading decentralised cloud storage provider dedicated to the Filecoin network based in Sydney. DSS operate enterprise scale compute and storage infrastructure in Tier 3 data centres throughout Australia with clients spanning the globe.

What is the primary source of funding for this project?

DSS is funding the project.

What other projects/ecosystem stakeholders is this project associated with?

Client Allocation Request for: Victor Chang Cardiac Research Institute #1937

Use-case details

Describe the data being stored onto Filecoin

The data sets are the original outputs of scientific cardiac research.

Where was the data in this dataset sourced from?

The data sets have been created by large-scale scientific cardiac research.

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this. 

DSS do not currently have permission from the client to share the data publicly, although it has the full cooperation from the client to verify data with notaries directly.

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

The existing dataset is limited by patient consent and whilst it is deidentified data internal policies and permissions do not currently allow for public use. Once DSS and the broader ecosystem have established a high degree of trust with VCCRI and its governance committees we seek to work with them to enable publicly available datasets that may be of value to the medical research community.

What is the expected retrieval frequency for this data?

The principle use case for the client is archival, thus retrieval is likely limited to twice a year.

For how long do you plan to keep this dataset stored on Filecoin?

Indefinitely

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

The storage deals will be distributed among at least four unique geographies. Certain elements of the data have sovereignty requirements, thus these will be limited to distribution within Australian territories. It is DSSs objective to distributed the datasets amongst the USA and Europe to the extent permissible by the client.

How will you be distributing your data to storage providers? Is there an offline data transfer process?

Online deals using Singularity.

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

DSS intend distributed data among SPs of enterprise scale with similar sealing capacity and whom operate tier 3 data centres.

How will you be distributing deals across storage providers?

Data that has a sovereignty requirement is intended to be distributed among DSS, Digital Income Fund, Holon and Vigilant IT. Datasets without a sovereignty may well be distributed among peers in other geographies, as these discreet datasets are identified by the client we will engage other SPs in the US and EU.

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

Yes, we have the resources/funding to begin making deals once we receive DataCap. 

We currently have the support we need thanks to the help of the Foundation, PL and other members of the community over the last few months.
marshyonline commented 2 years ago

Hi @swatchliu and @cryptowhizzard - Can we get the next trance signed ASAP? We are looking to get the next round of deals out ASAP - they are just waiting on this DC

mjroddy commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceag52ckd3t75r6afyl3qyisaeuz5ioiptm2n2yzghfdymmfpk7ujc

Address

f3qwluincblkdog6jovdcrv3yqqrlgxipnwv43un2iwbrofv63g6fmqogapwi3cf3fh4l3mdcrgtmfpbfphypa

Datacap Allocated

200.00TiB

Signer Address

f1ystxl2ootvpirpa7ebgwl7vlhwkbx2r4zjxwe5i

Id

c563fb4c-8ce6-480f-9f9b-e2c4ce4905eb

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceag52ckd3t75r6afyl3qyisaeuz5ioiptm2n2yzghfdymmfpk7ujc

kernelogic commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebfzv64wgngbw55t3gsphshhcbmcr32sv2t62jwlbskmm3yhtostm

Address

f3qwluincblkdog6jovdcrv3yqqrlgxipnwv43un2iwbrofv63g6fmqogapwi3cf3fh4l3mdcrgtmfpbfphypa

Datacap Allocated

200.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

c563fb4c-8ce6-480f-9f9b-e2c4ce4905eb

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebfzv64wgngbw55t3gsphshhcbmcr32sv2t62jwlbskmm3yhtostm

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f01885534

Client address

f3qwluincblkdog6jovdcrv3yqqrlgxipnwv43un2iwbrofv63g6fmqogapwi3cf3fh4l3mdcrgtmfpbfphypa

DataCap allocation requested

400TiB

Id

d3b63727-1d3b-42f1-8813-829eb2c9c6a0

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01885534

Client address

f3qwluincblkdog6jovdcrv3yqqrlgxipnwv43un2iwbrofv63g6fmqogapwi3cf3fh4l3mdcrgtmfpbfphypa

Last two approvers

kernelogic & megtei

Rule to calculate the allocation request amount

400% of weekly dc amount requested

DataCap allocation requested

400TiB

Total DataCap granted for client so far

350TiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

4.65PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
4711 6 200TiB 21.64 48.80TiB
cryptowhizzard commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebhh6nwuita25jdd3dqvsbvlyycifptvspvyjd4gp3dtc6q5majoi

Address

f3qwluincblkdog6jovdcrv3yqqrlgxipnwv43un2iwbrofv63g6fmqogapwi3cf3fh4l3mdcrgtmfpbfphypa

Datacap Allocated

400.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

d3b63727-1d3b-42f1-8813-829eb2c9c6a0

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebhh6nwuita25jdd3dqvsbvlyycifptvspvyjd4gp3dtc6q5majoi

DSS-AL commented 1 year ago

Hi all, can we please request that someone approve this tranche of datacap so that we can keep ingesting data? Much appreciated.

Fei Yan / Kernelogic / @kernelogic Wijnand Schouten / Speedium / @cryptowhizzard Eric / ByteBase / @swatchliu Mark Roddy / Holon / @mjroddy Meg Dennis / @MegTei Cabrina Huang / @xingjitansuo

mjroddy commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedaajd6whka6uupjnp3bgplphjktnjbrvixyad2e42o7zkcjavh2g

Address

f3qwluincblkdog6jovdcrv3yqqrlgxipnwv43un2iwbrofv63g6fmqogapwi3cf3fh4l3mdcrgtmfpbfphypa

Datacap Allocated

400.00TiB

Signer Address

f1ystxl2ootvpirpa7ebgwl7vlhwkbx2r4zjxwe5i

Id

d3b63727-1d3b-42f1-8813-829eb2c9c6a0

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedaajd6whka6uupjnp3bgplphjktnjbrvixyad2e42o7zkcjavh2g

mjroddy commented 1 year ago

Hi all, can we please request that someone approve this tranche of datacap so that we can keep ingesting data? Much appreciated.

Hi Andrew, all ok from here - sorry on delay.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f01885534

Client address

f3qwluincblkdog6jovdcrv3yqqrlgxipnwv43un2iwbrofv63g6fmqogapwi3cf3fh4l3mdcrgtmfpbfphypa

DataCap allocation requested

800TiB

Id

82f4dfa9-127a-4e9d-a72f-01ac4805146e

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01885534

Client address

f3qwluincblkdog6jovdcrv3yqqrlgxipnwv43un2iwbrofv63g6fmqogapwi3cf3fh4l3mdcrgtmfpbfphypa

Last two approvers

megtei & cryptowhizzard

Rule to calculate the allocation request amount

800% of weekly dc amount requested

DataCap allocation requested

800TiB

Total DataCap granted for client so far

350TiB

Datacap to be granted to reach the total amount requested by the client (5 PiB)

4.65PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
10240 6 400TiB 24.73 43.47TiB
filplus-checker commented 1 year ago

DataCap and CID Checker Report[^1]

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

✔️ Storage provider distribution looks healthy.

Provider Location Total Deals Sealed Percentage Unique Data Duplicate Deals
f01919423 Sydney, New South Wales, AU 88.95 TiB 23.90% 87.64 TiB 1.48%
f01319368new Sydney, New South Wales, AU 88.79 TiB 23.85% 86.48 TiB 2.60%
f01896422 Fremont, California, US 82.39 TiB 22.13% 80.89 TiB 1.82%
f01938357new Sydney, New South Wales, AU 78.01 TiB 20.96% 77.38 TiB 0.80%
f01156538 Sydney, New South Wales, AU 27.54 TiB 7.40% 26.91 TiB 2.27%
f01864434 Sydney, New South Wales, AU 3.29 TiB 0.88% 3.29 TiB 0.00%
f01206408 Sydney, New South Wales, AU 3.29 TiB 0.88% 3.29 TiB 0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

✔️ Data replication looks healthy.

Unique Data Size Total Deals Made Number of Providers Deal Percentage
1.69 TiB 1.69 TiB 1 0.45%
6.19 TiB 12.44 TiB 2 3.34%
2.25 TiB 6.88 TiB 3 1.85%
48.88 TiB 198.44 TiB 4 53.31%
29.75 TiB 152.03 TiB 5 40.84%
132.00 GiB 792.00 GiB 6 0.21%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

cryptowhizzard commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceb2hhzdlz4wsk4uztcggs3dyjlm4s7rd3t2zvziuevbzkipzxxnr2

Address

f3qwluincblkdog6jovdcrv3yqqrlgxipnwv43un2iwbrofv63g6fmqogapwi3cf3fh4l3mdcrgtmfpbfphypa

Datacap Allocated

800.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

82f4dfa9-127a-4e9d-a72f-01ac4805146e

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceb2hhzdlz4wsk4uztcggs3dyjlm4s7rd3t2zvziuevbzkipzxxnr2

NiwanDao commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebatiumj6cefhugs567nke6tu7zeda5x65ft5lx7q42ry4esz5ltw

Address

f3qwluincblkdog6jovdcrv3yqqrlgxipnwv43un2iwbrofv63g6fmqogapwi3cf3fh4l3mdcrgtmfpbfphypa

Datacap Allocated

800.00TiB

Signer Address

f1a2lia2cwwekeubwo4nppt4v4vebxs2frozarz3q

Id

82f4dfa9-127a-4e9d-a72f-01ac4805146e

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebatiumj6cefhugs567nke6tu7zeda5x65ft5lx7q42ry4esz5ltw

BDEio commented 1 year ago

@DSS-AL Hi! Great to see that you have gotten approval for DataCap! BDE is a verified deals auction house helping you to get paid storing your data with reliable storage providers. If you need any help, please get in touch.

marshyonline commented 1 year ago

checker:manualTrigger

cryptowhizzard commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the full report.

marshyonline commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

marshyonline commented 1 year ago

This Project is still underway and should be re-opened

Sunnyiscoming commented 1 year ago

Hello, @DSS-AL per the https://github.com/filecoin-project/notary-governance/issues/922 for Open, Public Dataset applicants, please complete the following Fil+ registration form to identify yourself as the applicant and also please add the contact information of the SP entities you are working with to store copies of the data.

This information will be reviewed by Fil+ Governance team to confirm validity and then the application will be allowed to move forward for additional notary review.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

-- Commented by Stale Bot.