Project: MOC Proof-of-Concept

cmbz commented 11 months ago

Purpose

Deploy containerized Dataverse on Mass Open Cloud to support large data and computing
Present demo of proof-of-concept at the Mass Open Cloud Alliance Conference on 2024/02/28

Background

Mass Open Cloud Alliance is a "partnership between higher education, medical research centers, government, and industry provides structure and resources that enable a close collaboration between research, development, and operations in a series of interconnected projects."
The MOC team wants to test a containerized Dataverse installation on the MOC infrastructure to support large data and computing.

Participants

Leonid Andreev, @landreev
Gustavo Durand, @scolapasta
Sonia Barbosa, @sbarbosadataverse
Ceilyn Boyd, @cmbz
Stefano Iacus, @siacus
Max Shad
James Culbert
Milson Munakami
Jim Myers, @qqmyers
Milan Kupcevic

Timeline

Start: 2024/01/08
- Milestone: Deliver containerized Dataverse installation (2024/01/15)
- Milestone: Binder equivalent functionality implemented (2024/01/31)
- Upload presentation (2024/02/23)
- Milestone: Presentation (2024/02/24)
- Close out tasks
End: 2024/04/01

Tasks

[x] Create NERC project (@siacus)
[x] Project management: @cmbz and @scolapasta and @siacus coordinate re. resource allocation
- [x] Assemble team
- [x] GEOS-CHEM use case (see https://github.com/IQSS/dataverse-pm/issues/154)
- [x] Set up recurring meetings, if needed (Slack channel used instead: https://iqss.slack.com/archives/C06EXB41EE5)
[x] Set up Dataverse container/installation for MOC (DevOps, @landreev)
[x] Arrange for authentication support for Keycloak
[x] Move from Binder to local NERC infrastructure to run workflows (Development, @pdurbin)
[x] Arrange for support of NESE tape connection
[x] Finalize presentation (@siacus)

Issues

[x] https://github.com/IQSS/dataverse-pm/issues/172

Technical Resources

[x] NESE tape resources
[x] NESE disk resources

Project Resources

cmbz commented 9 months ago

2024/01/29: Updated with new task: https://github.com/IQSS/dataverse-pm/issues/172

landreev commented 9 months ago

A possible dependency: IQSS/dataverse#10302. I can make a PR very quickly if/when needed.

landreev commented 9 months ago

Just a quick status checklist on the NERC deployment side:

[x] Basic Dataverse instance created (a Docker deployment in an OpenStack VM)
[x] Container Store configured (2TB)
[x] Access to the s3 bucket from within or outside the cluster
- [x] specifically, Dataverse instance configured w/ s3 access credentials
- [x] upload into the bucket via Dataverse tested
- [x] DvUploader works like a charm (tested with the GEOS-Chem data files used by the presentation notebook; see below)
- [ ] uploads via the UI need to be re-tested
[x] The Computation node setup remains the major component that's still missing, so this is the main focus of the current effort
- [x] We got the Python notebook from GEOS-Chem. It's nowhere near ready to be deployable in a standalone OpenShift Jupyter pod as has been the plan so far. I will be working with the author. #154 will be used to track the effort.
- [x] Once we got the notebook deployment figured out, the "external tool" mechanism will need to be added to it to perform the basic initial interaction with the Dataverse for obtaining the storage locations of the files.
[x] NESE tape storage volume needs to be set up and configured on the Dataverse instance via Globus, as secondary storage for the demo
- [x] Created a dedicated Globus ID and had roles set up by NESE giving it access to the separate tape and disk pools, so I can configure the demo instance with these Globus credentials to facilitate data uploads to tape

landreev commented 9 months ago

edited the checklist to add the NESE tape volume.

landreev commented 8 months ago

Marked the "NESE tape" item as complete on the checklist - under the assumption that it is complete "for the purposes of the MOC PoC presentation slides". I'm opening a new issue dedicated to figuring out a working setup, with a functioning Borealis app hosted on the server, etc., that can be used by real users, that we can later replicate in IQSS prod.

cmbz commented 8 months ago

2024/03/01

Closing issue as complete
See follow up: https://github.com/IQSS/dataverse-pm/issues/184

IQSS / dataverse-pm