GREI 5: HDV Task - Large Data Support

[cmbz]

Overview

Support the sharing of very large datasets (>TBs) by integrating the metadata in the repository with the data in the research computing storage" (Source: NIH OTA)

Tasks

• [x] https://github.com/IQSS/dataverse-pm/issues/105
• [x] https://github.com/IQSS/dataverse-pm/issues/106
• [ ] https://github.com/IQSS/dataverse-pm/issues/178 The following original NIH-OTA tasks are part of the Large Data Support proposal and pilot
  • Define a cost recovery model for large dataset support [was Aim 1.1.2]
  • MVP for integration the repository with computation (NERC) [was Aim 2.1.2] (See: https://github.com/IQSS/dataverse-pm/issues/201 and https://github.com/IQSS/dataverse-pm/issues/105)
  • Expand support for very large datasets through NESE and NERC; implement cost recovery model [was Aim 3.1.1]
  • Assess cost recovery model [was Aim 4.1.1]
• [ ] #243
  • [ ] #244
  • [ ] GREI 5 HDV - Task 2: Investigate approaches for determining costs of large data/compute services (Year 3)
  • [ ] GREI 5 HDV - Task 3: Investigate authentication approaches for specific large data/compute configurations based on use cases (Year 3)
  • [ ] GREI 5 HDV - Task 4: Design UI/UX for datasets or collections that use large data/compute resources (Year 3 and/or 4)

Issues

Pending

• [ ] https://github.com/IQSS/dataverse.harvard.edu/issues/260
• [ ] https://github.com/IQSS/dataverse.harvard.edu/pull/268

  Completed

• [x] https://github.com/IQSS/dataverse.harvard.edu/issues/261
• [x] https://github.com/IQSS/dataverse/pull/10451

Resources

• https://github.com/IQSS/dataverse-pm/issues/124
• APPROVED - NIH GREI Coopetition Plan Year 3

[cmbz]

Status: January 2024

Several Globus improvements and bug fixes were made to support large data deposits and integration with research computing services such as the Northeast Storage Exchange (NESE).

Completed

• https://github.com/IQSS/dataverse/pull/10201
• https://github.com/IQSS/dataverse/issues/10200

[cmbz]

Status: February 2024

A containerized Dataverse was deployed on Mass Open Cloud (MOC), using Northeast Storage Exchange (NESE) compute and resources to demonstrate how Dataverse can support computing on large Dataverse datasets stored on NESE tape resources. A demo of the proof-of-concept was presented at the Mass Open Cloud Alliance Conference on 2024/02/28. Work has begun to fully operationalize the strategy in Epic: Operationalize Large Data and Compute Infrastructure.

Completed

• https://github.com/IQSS/dataverse-pm/issues/105

[cmbz]

Status: April 2024

Large Data Support Working Group

• Zulip communications channel for Large Data Support Working Group established
  • Related: Large Data Support Breakout Session, DCM 2024
• Request made for working group page on GDCC website
• Current Working Group Members roster

Large Data Support Pilot

• The Oregon-Massachusetts Mammography Database (OMAMA-DB) was completely uploaded (not yet published). 7.9 TB for the large data portion, and 865GB for the smaller dataset. also see https://github.com/IQSS/dataverse-HDV-Curation/issues/443
• In progress is supporting a 16TB project from an MIT affiliate (end of May deadline) https://github.com/IQSS/dataverse-HDV-Curation/issues/444

[cmbz]

Status: May 2024

• Followed up on Geos-Chem to inquire about the PDF files in this collection.
• Tested Globus connection and NESE download of one file from OMAMA, successfully
• Demo Large data dataset will be created by @landreev so we can test download by non-Harvard affiliates, to help write instructions for data download
• See other updates and details here: https://github.com/IQSS/dataverse-pm/issues/178#issuecomment-2123402723
• Added two projects to the new section on "under consideration for support" in Pilot Large Data Issue

[cmbz]

Status: June 2024

• TBD