Run `ClimaAtmos` on `Google Cloud`

[sriharshakandala]

The Climate Modeling Alliance

Software Design Issue 📜

Purpose

Setup an automated build system to deploy ClimaAtmos.jl on the google cloud.

Cost/Benefits/Risks

Benefits:

• The approach can be generalized to other cloud services and supercomputers.
• Provides access to nodes having up to (16) GPUs for benchmarking scaling performance.

People and Personnel

• Lead: @sriharshakandala , @charleskawczynski
• Collaborators:
• Reviewers: -->

Components

• Setup an automated build system for running ClimaAtmos on Google Cloud
• Benchmark scaling performance on up to (16) GPUs on a single node

  Inputs

Results and Deliverables

Task Breakdown And Schedule

SDI Revision Log

CC

@tapios @simonbyrne @cmbengue

### Tasks
- [x] Install/configure and run `ClimaAtmos.jl` on a single GPU on GCP
- [x] Install/configure SLURM for single node and multi-node GCP configurations
- [x] Setup environment for CliMA software stack
- [ ] Compile strong scaling data on H100 GPUs
- [ ] Add curated documentation to ClimaDocs

[Sbozzolo]

Would it be possible to have more details on this SDI?

I am interested in potential sinergies to improve the resiliance of our CI infrastructure

[charleskawczynski]

I am interested in potential sinergies to improve the resiliance of our CI infrastructure

We're still in the process of adding details. I think we'll want to keep this decoupled, to start with, from the Central CI infra, as the central configuration is more static than what we'll be doing on GCP.

[Sbozzolo]

I am interested in potential sinergies to improve the resiliance of our CI infrastructure

We're still in the process of adding details. I think we'll want to keep this decoupled, to start with, from the Central CI infra, as the central configuration is more static than what we'll be doing on GCP.

Okay, thank you! I am particularly interested in systems to check/ensure that the fastest trasport protocol is used for MPI communication with GPUs, which is something that needs to be configured and checked on every machine and with different combinations of libraries. If you build something to check that you are doing that on GCP, I think that the same solution might be adapted more generally. So far, I had to resort manual labor.

Also, I set up and configured spack when we first got clima. Eventually, we decided to not use it because Scott installed the libraries we needed systemwide with the correct build options.

I am attaching the environment I set up at the time in case you might find useful. (This is several months old, so it might not be relevant)

spack.yml.txt

[Sbozzolo]

Leaving this question for when time comes:

How are you planning on handling larger input files (as the ones in ClimaArtifacts)?