Automated testing

[TomNicholas]

We have some tests, but we want them to run regularly (e.g. on every pull request).

The conventional way to do this is to use github actions. This is a service which spins up one or more virtual machines in the cloud on demand to run your test code. It's configured using special .yaml files that would live in this repository. We likely don't have to pay to use GH actions (as this is a public repository, and assuming we aren't trying to run tests suites that take longer than the 30 minute limit on their free tier).

The only tricky bit of getting these automated tests running is the need for compiling the fortran code, and the desire for potentially using commercial compilers that have a restrictive license. However, MARBL apparently already has something like that set up that we can hopefully copy. (@dafyddstephenson do you have a link?)

I'm happy to help get this set up but will need to work with someone who knows more about the specifics of the compiling.

[TomNicholas]

We also don't have to let the perfect be the enemy of the good with this. I expect that even running the tests we already have with a single compiler (e.g. a free one like gfortran) automatically would already be very helpful, and provide a starting point to build on.

[matt-long]

MARBL has a test suite here. It does not use GitHub Actions (but that's an implementation detail). It does compile a standalone MARBL driver with multiple compilers, so we can refer to this as a template to help design the test suite for ROMS.

[dafyddstephenson]

Thanks for taking the initiative on this @TomNicholas. I'm not sure how recently, but MARBL now does have automated testing; the workflow is here.

I agree that if we could get any form of this running, that would be a great start. I'm already compiling with gfortran, so if that's all we're limited to it shouldn't be too restrictive.

[TomNicholas]

It does not use GitHub Actions (but that's an implementation detail)

Looks like it uses Travis CI, which is a 3rd-party service that does basically the same thing as Github Actions. So yeah the MARBL testing is automated, so this looks like a great place to start.

[TomNicholas]

It looks like the MARBL testing works via this .yml file that specifies that this bash script run_test_suite.sh should run. We could start by creating something a similarly simple script for ROMS (if it doesn't exist already) that we can call from within a github actions job.

[TomNicholas]

@dafyddstephenson We need to add a workflow file that looks something like this:

name: Automated Testing

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - '*'

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout Repository
      uses: actions/checkout@v2
      with:
        repository: <your-repository-url>
        ref: main

    - name: Set up Conda
      uses: conda-incubator/setup-miniconda@v2
      with:
        environment-file: path/to/your/environment.yml

    - name: Create Conda Environment
      run: conda env create -f path/to/your/environment.yml
      shell: bash

    - name: Activate Conda Environment
      run: source activate <your-conda-environment>

    - name: Set Environment Variables
      run: source your-script.sh
      shell: bash

    - name: Compile Fortran Code
      run: make -C path/to/your/fortran/code
      shell: bash

    - name: Run Tests
      run: bash path/to/your/test-script.sh
      shell: bash

this needs to go in ucla-roms/github/workflows/test_workflow.yml

[dafyddstephenson]

Just as an update I've been working on this and am close to an automated workflow using gfortran that runs the existing test suite (in Examples/code_check) on push to main. Can see progress on my fork (specifically the roms-testing branch). Should be ready in a few days.

[dafyddstephenson]

Ok, I've got it going over on my fork. Right now it's using gfortran and running $ROMS_ROOT/Examples/code_check/do_test_all.sh which runs the Pipes_ana,Pipes_real,Rivers_ana,Rivers_real,Flux_frc, and Filament configurations and compares 4 metrics in the output log with those in a "benchmark" output log. I've created new benchmark files for the automated testing using GitHub codespaces so now these are used for comparison on push (they are now passing).

There is some substantial tidying up to do as I've done some scrappy stuff just to get it off the ground, and the gfortran issue persists. @nmolem, could you please push the changes you've made that make ROMS gfortran friendly? I'll then make a branch off the latest main, tidy up the automated testing implementation, and issue a PR.

Thanks all!

[nmolem]

I just did. On a branch named 'compliant', for you to try.