search

ioos / Cloud-Sandbox

IOOS' Coastal Modeling Cloud Sandbox provides a framework for developing, modifying and running models in the cloud. It provides repeatable configurations, model code and required libraries, input data and analysis of model outputs. The Sandbox supports not only the development of services and models, but also Cloud HPC to run and validate models.
https://www-sandbox.ioos.us/
BSD 3-Clause "New" or "Revised" License
15 stars 12 forks source link

A feature enhancement is proposed: Stand up a process for generating model containers #91

Open KatherinePowell-NOAA opened 3 months ago

KatherinePowell-NOAA commented 3 months ago

Model software containerization.

Current state: "running a model" or "doing a run" is actually a build step and a compile step. There should be a possibility of a "buildtime" machine and a "runtime" machine.

This is what happens with cloudflow, in that the head node appears to be the "build" machine (and the scheduler) and the worker nodes of the cluster execute things as a runtime action. This can (and should) be broken out by the time Mykel is finished with the containerization.

KatherinePowell-NOAA commented 3 months ago

Is it at all possible that we could strip down the head node to be a runtime-only node and purpose-build a machine as a container to perform compilation as part of a CI process rather than as part of "running the model"? Does more compilation happen after the run?

If we did both of those things, then the head node could be an unconfigured basic image that we could do whatever needed to be done with it. That also means that we would need to directly replicate the head node and any relevant worker nodes as containers (i.e. what I'm trying to do these days) such that a non-CAC user could make use of for a docker-compose. It would also mean that the compilation process might be workable inside something like Github.

mykelalvis commented 3 months ago

I cannot be assigned here.

KatherinePowell-NOAA commented 1 month ago

Mykel added documentation for model generation in the sandbox (internal draft to be shared later), and specific model configurations and dependencies for CORA/ADCIRC after 7/30/2024 meeting with John Ratcliff.

breannalcv commented 1 month ago

Stand Up Model Containers