filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] Speedium - NIH NCBI Sequence Read Archive [NON FIL-E] #2295

Open cryptowhizzard opened 11 months ago

cryptowhizzard commented 11 months ago

Data Owner Name

NIH - National Institute of Health

What is your role related to the dataset

Data Preparer

Data Owner Country/Region

United States

Data Owner Industry

Life Science / Healthcare

Website

https://www.nih.gov/

Social Media

https://www.facebook.com/nih.gov/

Total amount of DataCap being requested

15PiB

Expected size of single dataset (one copy)

13.14 PiB

Number of replicas to store

10

Weekly allocation of DataCap requested

1PiB

On-chain address for first allocation

f1k22mdjvcqhuvrflxrftp4f5oyvgzvnt6rr3kyyi

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Identifier

No response

Share a brief history of your project and organization

Since its launch, the Filecoin network has become an important player in the decentralised storage space, offering a secure and transparent alternative to traditional data storage solutions.

We as Speedium / DCENT have been engaged with storing real and valuable datasets on the Filecoin network since Slingshot 2.6 and have been actively developing tools to improve the process. We are always on the lookout for new and useful client data to onboard.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

NIH NCBI Sequence Read Archive (SRA) on AWS
The Sequence Read Archive (SRA), produced by the [National Center for Biotechnology Information (NCBI)](https://www.ncbi.nlm.nih.gov/) at the [National Library of Medicine (NLM)](http://nlm.nih.gov/) at the [National Institutes of Health (NIH)](http://www.nih.gov/), stores raw DNA sequencing data and alignment information from high-throughput sequencing platforms.

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

No response

If you are a data preparer. What is your location (City and Country)

Our datacenter is situated in Heerhugowaard, a town located just north of Amsterdam in the Netherlands.

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

The dataset's information originates from Amazon AWS, and it is processed using the Singularity software. This specialized tool is designed to cater to large-scale clients who require onboarding of PB-scale data onto the Filecoin network. We have allocated a substantial pool capacity of 13 PiB to securely store a complete copy of this dataset until it is fully distributed. Once the distribution is complete, we will provide hot copies to entities involved in repairing or restoring the dataset.

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

This dataset has not been previously stored; we are the pioneering entity attempting to do so.

Please share a sample of the data

https://registry.opendata.aws/ncbi-sra/

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Monthly

For how long do you plan to keep this dataset stored on Filecoin

1.5 to 2 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, North America, Europe, Australia (continent)

How will you be distributing your data to storage providers

HTTP or FTP server, IPFS, Lotus built-in data transfer

How do you plan to choose storage providers

Slack, Partners

If you answered "Others" in the previous question, what is the tool or platform you plan to use

No response

If you already have a list of storage providers to work with, fill out their names and provider IDs below

| MinerID | City | Continent | Business/Entity |
| --- | --- | --- | --- |
| `f01944347` | Oregon | USA | Jenny, Dabai |
| `f01952350` | Oregon | USA | Jenny, Dabai |
| `f01972364` | Oregon | USA | Jenny, Dabai |
| `f01972376` | Oregon | USA | Jenny, Dabai |
| `f02000937` | Chengdu | CN | MTY |
| `f01915033` | Chengdu | CN | MTY |
| `f0120****` | Melbourne | AU | HOLON |
| `f0115****` | Melbourne | AU | HOLON |
| `f01199430` | Heerhugowaard | EU | DCENT |
| `f01786387` | Heerhugowaard | EU | DCENT |
| `f01201327` | Heerhugowaard | EU | DCENT |
| `f01937642` | Heerhugowaard | EU | DCENT |
| `f0198****` | Dallas | USA | GREATERHEAT |
| `f0188****` | Singapore | AS | GREATERHEAT |
| `f01091851` | Omaha | USA | DLTx |
| `f01736668` | Omaha | USA | DLTx |
| `f01820744` | Omaha | USA | DLTx |
| `f0855584` | Omaha | USA | DLTx |
| `f01794610` | Omaha | USA | DLTx |
| `f01838599` | Kansas City | USA | DLTx |
| `f01845552` | Kansas City | USA | DLTx |
| `f01274011` | Sydney | AU | DSS |
| `f01746964` | Sydney | AU | DSS |

How do you plan to make deals to your storage providers

Boost client

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

aggregation-and-compliance-bot[bot] commented 9 months ago
Client f02897439 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 21 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Shared data percent < 20% 26.26% of the clients data is shared with other clients. This should be less than 20%
aggregation-and-compliance-bot[bot] commented 8 months ago
Client f02897439 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 28 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Shared data percent < 20% 27.44% of the clients data is shared with other clients. This should be less than 20%
aggregation-and-compliance-bot[bot] commented 8 months ago
Client f02897439 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 36 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Shared data percent < 20% 29.28% of the clients data is shared with other clients. This should be less than 20%
large-datacap-requests[bot] commented 8 months ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f02049625

Client address

f1k22mdjvcqhuvrflxrftp4f5oyvgzvnt6rr3kyyi

DataCap allocation requested

2PiB

Id

96ea69ce-0e59-4ac8-929b-728a64be6f63

aggregation-and-compliance-bot[bot] commented 8 months ago
Client f02897439 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 43 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Shared data percent < 20% 30.66% of the clients data is shared with other clients. This should be less than 20%
aggregation-and-compliance-bot[bot] commented 8 months ago
Client f02897439 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 50 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Shared data percent < 20% 29.99% of the clients data is shared with other clients. This should be less than 20%
aggregation-and-compliance-bot[bot] commented 7 months ago
Client f02897439 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 58 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Shared data percent < 20% 29.35% of the clients data is shared with other clients. This should be less than 20%
aggregation-and-compliance-bot[bot] commented 7 months ago
Client f02897439 does not follow the datacap usage rules. More info here. This application has been failing the requirements for 65 days. Please take appropiate action to fix the following DataCap usage problems. Criteria Treshold Reason
Shared data percent < 20% 28.6% of the clients data is shared with other clients. This should be less than 20%