search

ioos / Cloud-Sandbox

IOOS' Coastal Modeling Cloud Sandbox provides a framework for developing, modifying and running models in the cloud. It provides repeatable configurations, model code and required libraries, input data and analysis of model outputs. The Sandbox supports not only the development of services and models, but also Cloud HPC to run and validate models.
https://www-sandbox.ioos.us/
BSD 3-Clause "New" or "Revised" License
15 stars 12 forks source link

Document requirements and Implement model run(s) for CO-OPS #79

Open Michael-Lalime opened 5 months ago

Michael-Lalime commented 5 months ago

@ZacharyWills @KatherinePowell-NOAA

We received requirements from CO-OPS regarding needs for a development space. I want to ask your advice about setting up a space or possibly even a separate head node for them.

What do you think? Can we make something like this work in the near-ish future?

Problem Statement CO-OPS currently has two AWS servers for OFS applications; ofs-east1 is an operational server to generate all OFS operational products; ofs-work-east1 is presently used as a developmental server for software and product development and testing, and to create real-time products for all OFS which are run in a semi-operational mode for development and testing. Therefore, ofs-work-east1 is mainly used as a parallel/staging server. There is an urgent need to build a new developmental AWS server that the modeling team can use for OFS software and product development and testing.

This document contains requirements for ofs-dev-east1 to support CO-OPS development work. Any requests made herein to increase license count, CPU, etc., will need to be reviewed and/or approved and will impact the system's delivery date. Requirements Hardware requirements (Same hardware as ofs-work-east1, number of CPUs, memory, etc.) Modeling team (9 modelers) heavily relies on ofs-dev OS Change For awareness the OS will change from RHEL 7 which is running on ofs-work-east1 and ofs-east1 to RHEL 8 Disk space 5TB EBS for /opt/archive (software, and semi-operational OFS under development) 8 TB EBS mount as /opt/ofsdev/users - used by modelers for R&D related projects, such as testing, developing, skill assessment etc. +space exclusive to the SCI/SA project What models will run on this server? Only OFS under testing is run on dev server, no semi-operational OFS runs on dev using cron jobs. Mount /opt/ofs-bdp and /opt/ofs-all S3 buckets for data access (READ ONLY) Other NOAA s3 bucket of data sources (e.g. satellite data, NCEI archives, etc.) might be needed to be mounted on. It can be figured out as needed in the future. 1TB /cache drive This is required for the requested s3fs mounts

List of Software and Compilers Matlab (1 concurrent license) IDL (1 license) ifort Fortran compiler Icc C compiler Totalview or similar debugger Eclipse software development kit Java NCL command Emacs GIT () Anaconda with Python 3.9 or newer, Jupyter Notebook & Library NCO NetCDF operators Latest version NetCDF C and Fortran NetCDF commands (e.g. ncdump) Xmgrace xmgredit5 Special Needs: commands saved in /usr/bin, nedit, gedit, gthumb Possibly Allow CO-OPS external users to access the dev server co-development with partners critical for closing the O2R cycle

OFS Data Processing System (SCI/SA) Requirements Compute

Storage/Data: Local disk storage (DEV possibly 1TB - it is covered by 8 TB under /opt/ofsdev/users ) Need 2 years of historical OFS data. Historical OFS data at the NODD arn:aws:s3:::noaa-nos-ofs-pds /opt/ofs-bdp will be mounted for READ ONLY Note the SCI/SA project requires NODD S3 access to various data sets, including STOFS-2D-Global, STOFS-3DAtlantic, GFS, HRRR, etc.)

Software:

Apache Tomcat Python 3 (3.9 or newer) Anaconda and Pipenv

mwengren commented 5 months ago

I think some further discussion about CO-OPS needs/goals would be helpful. These requirements, for me at least, bring up the question of scope of the Sandbox, because I think these are broader than what we initially planned for.

It may be that some aspects belong in the Sandbox, whereas others do not.

Also, I don't see in these two descriptions the need to actually run ocean models, which is our primary mission:

I don't know how best to have this conversation. Perhaps we can do it here in this GitHub issue. I think once we can get a feel on the scope question, it may make sense to break these requirements down into more manageable pieces to implement the parts we feel belong in the Sandbox.