Project: Pilot Large Data Support Service

cmbz commented 9 months ago

Overview

Pilot Harvard Dataverse large data support services using NESE tape resources for several datasets from Harvard affiliates.

Tasks

[x] Identify prospective data collections
[x] Confirm pilot participants
[x] Coordinate with pilot collection owners to estimate collection size and curation needs
[x] Create Harvard Dataverse collections
[x] Coordinate with NESE support staff for tape provisioning
[x] Coordinate with data collection owners, NESE support, and Dataverse team to upload data
[x] Coordinate with IQSS finance re. costs
[ ] Assess pilot and propose workflow improvements

Process Development and Management

[x] Develop an intake process including basic and extended consultation process
[x] Develop and launch RT queue, including service attributes (will map to service offerings)
[x] Develop large data collection intake form @sbarbosadataverse
[ ] Define what is included in the Large Data Technical Support service component
[ ] Define what is included in the Large Data Administration service component
[ ] Define what is included in the Large Data Monitoring service component
[x] End-user instructions for NESE data access, add to Dataverse documentation in Guide and HDV support website
[ ] Document large data curation services process (see hdv-curation issues on large data)

Team

@sbarbosadataverse
@cmbz
@qqmyers
@landreev

Pilot Participants

School of Engineering and Applied Sciences (Harvard SEAS)

https://github.com/IQSS/dataverse-HDV-Curation/issues/594

Terabyte-Scale Dataset for Partial Discharge 2 Detection in Covered Conductors via Contact 3 Galvanic Method

-https://db.cs.vsb.cz/datacolls/pdp2020/ -https://help.hmdc.harvard.edu/Ticket/Display.html?id=366158&results=56072ee681951385a6015bad30361ae4

The Oregon-Massachusetts Mammography Database (OMAMA-DB)

Size: 8T
Contact: Daniel Haehn, SEAS & UMass Boston

GEOS-Chem 1 and 10 year Benchmark data

Datasets
- http://ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/
- http://ftp.as.harvard.edu/gcgrid/geos-chem/10yr_benchmark
- Size: about 1.6 TB:
- 484G 10yr_benchmarks/
- 484G total
- 1.1T 1yr_benchmarks/
- 1.1T total
- Note: It would increase as they add new benchmark output. typically release model versions once per quarter. Prior to each release they would generate 1 or more benchmark runs for GCClassic and GCHP. The 10 year benchmarks are generated once per major version release which are less frequent.

DrivAerNet: A Parametric Car Dataset for Data-driven Aerodynamic Design and Graph-Based Drag Prediction

[x] moving their ‘streamlined’ dataset (<1TB) to the DeCoDE lab’s dataverse
<1TBish
File sizes: Largest: 1.6G and smallest: 400bytes,
Total number of files: not sure, user quoted 10^6, but was fuzzy on that.
File types: .tar.gz, txt, vtk, stl, .sh, data will be packed in .zip format
[x] By mid to end of May: the full 16TB.

Under discussion for support

-https://help.hmdc.harvard.edu/Ticket/Display.html?id=362556 -https://help.hmdc.harvard.edu/Ticket/Display.html?id=358471

Related RT Tickets

Issues

Resources

Google Drive Folder

cmbz commented 7 months ago

Status: March 2024

Our pilot for computing on large data was completed. See follow up: https://github.com/IQSS/dataverse-pm/issues/184 and continuing work here: https://github.com/IQSS/dataverse.harvard.edu/issues/254
Closed
https://github.com/IQSS/dataverse-pm/issues/105

cmbz commented 7 months ago

Status: April 2024

Improvements made to Globus: "The PR fixes the issue with multifile Globus transfers out/downloads not working for draft datasets. It also improves handling of cases where ineligible files are selected for download or Globus transfer by only showing the download/transfer mechanisms that will work (on some files) given the files in the dataset and by improving the UI messages to indicate that files may not be eligible either because the user doesn't have permission (restricted or embargoed), or because the files can't be downloaded/transferred (i.e. files not in a Globus store when the user tries a Globus transfer or files in a Globus store that doesn't support normal downloads when the user selects download.)"

https://github.com/IQSS/dataverse/pull/10451

sbarbosadataverse commented 6 months ago

Status: May 2024

Added new Process Development and Management section to this issue
Tested Globus/NESE download for OMAMA dataset, download successful but found that guestbook is not collecting downloads, could be due to the dataset being in "draft"
Leonid created a demo dataset for testing Globus access for non Harvard affiliates: https://demo.dataverse.org/dataset.xhtml?persistentId=doi:10.70122/FK2/URDDBC
Ceilyn revised documentation for large data services, budget planner, and internal HDV service components
New section added to record groups interested, but not yet confirmed for the pilot

cmbz commented 5 months ago

Status: June 2024

RT for large data support is created and populated (still require attributes we want to track)
Started budget for Jax labs https://help.hmdc.harvard.edu/Ticket/Display.html?id=363736
Started Globus instructions for users for OMAMA dataset

sbarbosadataverse commented 4 months ago

Status: July 2024

Draft intake form started
Instructions to download data from NESE complete
OMAMA collection updated with NESE download instructions and checklist for curation completed (minus Invoice): https://github.com/IQSS/dataverse-HDV-Curation/issues/443
- Estimate for https://github.com/IQSS/dataverse-HDV-Curation/issues/444 completed and shared with Author
- Connected with Bob for Geos-Chem and received instructions on preparing the data for both NESE and s3 storage. Waiting on his details about the PDFs in the collection.

cmbz commented 3 months ago

Status: August 2024

Related large data services updates in https://github.com/IQSS/dataverse-pm/issues/268#issuecomment-2243299175
Pending

sbarbosadataverse commented 1 month ago

Status: September 2024

Bertarelli is interested in 10TB of storage to NESE disk
Related issue: https://github.com/IQSS/dataverse.harvard.edu/issues/305

cmbz commented 3 weeks ago

Status: October 2024

Quote was prepared for Bertarelli Collections, standard pricing
Testing and troubleshooting of Globus upload instructions performed by MIT researcher and Dataverse team
Draft of Large Data Services intake Qualtrics form was developed and will be tested in November.

cmbz commented 2 weeks ago

Status: November 2024

Published: DrivAerNet: A Parametric Car Dataset for Data-driven Aerodynamic Design and Graph-Based Drag Prediction. Discussed how to invoice MIT-content (As affiliates) and the invoice process for this collaboration. This dataset may need to rework some data packaging as they did not follow the required stamdards for one of their datasets.
Our IT/RT Support team has completed testing the uploading and downloading via Globus and the scope of their work in assisting with these services has been entered into the services table, as requested.

IQSS / dataverse-pm