Science in the Cloud key elements

[rsignell-usgs]

Harry House asked me what the key elements would be for doing science in the cloud. Since these are what we are trying to do on HyTEST, I though I would try listing them here:

• Use flexible open-source frameworks already proven successful in the community, and adapt/enhance them for working effectively on the cloud. Pangeo didn't start from scratch -- it used the popular tools xarray, pandas, dask, Jupyter, holoviz and the NetCDF data model to create a powerful scalable, cloud-optimized environment for scientific analysis and visualization.
• Stay current with developments in cloud tech relevant to science. Participate in organizations like the ESIP Cloud Computing Cluster, the Pangeo Community and the OGC Testbeds.
• Explore new technologies without institutional hurdles. The ESIP Lab, supported by USGS, NOAA and NASA, provides a free cloud environment where any technology can be explored without the constraints of agency and institution cloud contracts. Once identified, critical elements of cloud technology can then be advocated at the institutional level.
• Deploy and maintain flexible, familiar frameworks that work on any cloud. For HPC, tools like AWS Parallel Cluster allow an environment to be deployed (or destroyed) in minutes that looks virtually identical to on-prem HPC: SLURM for scheduling, Lustre for parallel disk, module and Spack for software deployment. This allows nearly frictionless transition of workflows from on-prem to cloud. Other cloud providers have similar packages, tuned to make sure HPC is optimized for their cloud environment, but providing the same familiar user experience. For HTC, including data processing, analysis and visualization, tools like Nebari allow and environment to be deployed that use Kubernetes (Docker container orchestration) for scaling and portability, as well as a powerful JupyterHub and Dask Gateway environment, allowing the computations to meet any scientific need.
• Contract companies and individuals with proven cloud experience to build out the frameworks. Although science-driven institutions may occasionally get lucky and be able to hire someone with cloud experience, it's hard to keep these folks. Companies that specialize in this work can provide appropriate pay, support structure and environment that cater to their needs (and expectations).
• Make sure our data is cloud-optimized, and publish high-value data via object storage on a high-speed (100Gbe+) network. For USGS smaller datasets (say <100Gb) can be published on ScienceBase directly (currently AWS S3) as the storage and egress fees will be affordable. For larger datasets, USGS should publish to tape (currently BlackPearl) with a Globus endpoint for access. (The seminal dataset for this approach is the COAWST Forecast Model Archive, 14TB!). High-value datasets can then be derived and distributed by a data broker (e.g. partner or contractor) and distributed on a cloud provider public data program (if accepted) or on sites like the Open Storage Network (if not accepted). Open Storage Network is an NSF-funded project that allows data providers to purchase 1PB of object storage for about $100K. When this hardware is placed on a high-speed network (e.g. Internet 2), it delivers performant S3-compatible access to data with no egress fees, and if the lifetime is 5 years, at a fraction of the storage cost of commercial cloud providers.
• Use community standards for data models, metadata, catalog search and other APIs. The community has used the spreadsheet model for tabular data, the GDAL model for raster data, the shapefile model for vector data and the NetCDF model for ndimensional array data for more than two decades. While for data formats have come and gone, the data models remain. Tabular data can now be stored in cloud-optimized parquet format instead of CSV. Raster data can be stored at Cloud-Optimized Geotiff (but still be accessed using GDAL). Vector data can be stored as cloud-performant GeoParquet. Ndarray data can be stored using the Zarr format or using Kerchunk. Metadata can be stored as STAC items and searched using the STAC API. New cloud-performant backends like ZarrDAP and Xpublish can be written to support community standard data APIs like OPeNDAP.

[rviger-usgs]

@rsignell-usgs this is really timely. Just got poked to update our project plan at the end of yesterday and will be what I'm chewing on for today. Thanks!

[rsignell-usgs]

@rviger-usgs , cool!
cc @amsnyder

[rsignell-usgs]

Closing this since it's not really an open issue. Just wanted to capture these thoughts.