search

NOAA-EMC / UPP

Other
36 stars 98 forks source link

add unit testing to UPP? #846

Closed edwardhartnett closed 2 months ago

edwardhartnett commented 8 months ago

Currently, the CI jobs just build upp, they don't run any tests.

At one time we had a plan with Kyle taking the test code that @WenMeng-NOAA has in another repo, and combining it into UPP and using it for unit testing. Alas, Kyle is no longer with us, but it's still a good plan.

Unit testing is a fundamental software engineering technique of proven value. We have brought unit testing to many EMC codes, and they benefit with fewer bugs, faster development, and greater developer confidence.

Can we come up with a plan to add unit testing to UPP? The NCEPLIBS team, including myself and @AlexanderRichert-NOAA , are ready to help. What we need are test codes that can run and exit 0 for success, anything else for failure. Once we have that, we can have the CI run the tests automatically for every PR.

edwardhartnett commented 3 months ago

We have a meeting coming up about this topic - can we get some more info to prepare for it?

@WenMeng-NOAA are there test codes in another repo? If so, can you tell us where they are so we can take a look at them?

FernandoAndrade-NOAA commented 3 months ago

The meeting is to discuss what would be needed to introduce unit testing into UPP. From what I understand, there is no preexisting testing code present in another repo that would satisfy a setup for unit testing, so it would need to be created from scratch for UPP. I'm only aware of the regression tests we run on PRs before merging.

edwardhartnett commented 3 months ago

What are the regression tests that are run before merging? Can we have more info?

Perhaps these can be turned into tests run by the CI system automatically...

FernandoAndrade-NOAA commented 3 months ago

The RTs are run from UPP/ci/rt.sh which pulls in the test scripts for 8 test cases, each with an accompanying pe_test for a total of 16 which are placed within the workdir specified by rt.sh along with test output files. The test scripts for the test cases themselves are located in the default home directories listed within rt.sh: Hera: /scratch2/NAGAPE/epic/UPP/test_suite/jobs-dev Orion / Hercules: /work/noaa/epic/UPP/jobs-dev

FernandoAndrade-NOAA commented 3 months ago

Queue permitting, the RTs are usually complete within 15 minutes. If you'd like to run them with the develop branch:

git clone https://github.com/NOAA-EMC/UPP.git
cd UPP/ci
./rt.sh -a <account_number> -r <run_directory>

The work directory created within UPP/ci will contain the same job scripts from jobs-dev along with their output if anything failed abnormally (out of disk space, seg faults, etc). The run directory contains the data output along with any .diff files if a developer made any result-changing updates.

edwardhartnett commented 2 months ago

It's been decided not to proceed with this.

If anyone's interested, my working unit test is here: https://github.com/edwardhartnett/UPP/pull/1

The NCEPLIBS team will continue to provide the same high level of support to UPP as we always have, using the required manual processes. Thanks everyone for your attention.