Testsuite logging

[rs028]

Improving the logs of the various tests (see #423).

[codecov[bot]]

Codecov Report

Merging #482 (0257d2d) into master (6650b8a) will decrease coverage by 0.05%. The diff coverage is n/a.

Additional details and impacted files

[](https://app.codecov.io/gh/AtChem/AtChem2/pull/482?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AtChem) ```diff @@ Coverage Diff @@ ## master #482 +/- ## ========================================== - Coverage 66.81% 66.76% -0.05% ========================================== Files 17 17 Lines 2046 2046 ========================================== - Hits 1367 1366 -1 - Misses 679 680 +1 ``` | Flag | Coverage Δ | | |---|---|---| | build | `52.81% <ø> (ø)` | | | tests | `66.92% <ø> (-0.25%)` | :arrow_down: | | unittests | `54.86% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AtChem#carryforward-flags-in-the-pull-request-comment) to find out more. [see 1 file with indirect coverage changes](https://app.codecov.io/gh/AtChem/AtChem2/pull/482/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AtChem) ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/AtChem/AtChem2/pull/482?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AtChem). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AtChem) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/AtChem/AtChem2/pull/482?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AtChem). Last update [6650b8a...0257d2d](https://app.codecov.io/gh/AtChem/AtChem2/pull/482?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AtChem). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AtChem).

[rs028]

To avoid overcrowding the terminal, the idea here is to output to screen a message saying that the test has failed or passed at the end of all tests. Initally limited to indent and style tests only for simplicity. The initial submit works for individual tests (eg make indenttest) but not when they are put together (make alltests) because the temporary file gets deleted at the start of each test.

Desired output:

Running indent script on:
src/argparse.f90
src/atchem2.f90
src/atmosphereFunctions.f90
src/configFunctions.f90
src/constraintFunctions.f90
src/dataStructures.f90
src/inputFunctions.f90
src/interpolationFunctions.f90
src/outputFunctions.f90
src/parameterModules.f90
src/solarFunctions.f90
src/solverFunctions.f90
Indent script finished

Running style script on:
src/argparse.f90
src/atchem2.f90
src/atmosphereFunctions.f90
src/configFunctions.f90
src/constraintFunctions.f90
src/dataStructures.f90
src/inputFunctions.f90
src/interpolationFunctions.f90
src/outputFunctions.f90
src/parameterModules.f90
src/solarFunctions.f90
src/solverFunctions.f90
Style script finished

Indent test PASSED
Style test PASSED

[rs028]

A different approach is to log all the output of the test to a file and just output to screen a message. This is much simpler and cleaner. @spco, any thoughts?

[rs028]

This works for indent, style, model and old tests. ~The only annoying thing is that for the test spec_yes_env_no_with_jfac_fail1, which is meant to fail, the executable prints a STOP 2 message (I think) and it looks a bit untidy ;)~

[rs028]

I can't find a way to log the unit tests. If I understand it, it is all hard wired in the FRUIT code and I don't really want to modify that. But the printout to screen is actually reasonably clear, so I think this is okay.

[rs028]

@spco I think I have done the merge correctly, but for some reasons, it is failing the macos tests again! If you have some time to review this PR as well, it would be great :)

[rs028]

More detail about the tests.

Ubuntu:

• unittests, oldtests and modeltests pass on all versions

MacOS:

• run fails with gfortran 9
• unittests and oldtests pass
• 4 out of 7 modeltests fail

All systems report a warning about cmake becoming obsolete during the installation of sundials, but I don't think that is causing a problem here.

Another warning message is ld: warning: dylib (cvode/lib/libsundials_cvode.dylib) was built for newer macOS version (11.7) than being linked (11.0), I don't know if that is why the MacOS tests fail (?)

~In general the whole setup is a bit clunky now. I had to run each test manually and individually (sometimes more than once) to get it to complete the run. Perhaps it is all just a problem on the github actions side.~ (this is okay now that the yml file has been modified to run all tests, regardless of their success).