CLI testing for FRE utils

[cwhitlock-NOAA]

I'm starting a basic cli workflow to make sure that the command-line utilities (fre pp validate, etc.) will run. I'd like to expand the testing suite in the following steps:

1. does this run? (check status of $toolname --help)
2. does this run and produce correct output? (actually use the tool; compare against prior results)

1) is as much focused around building a functional cli workflow as making the tests themselves - with something that runs on push, we should be able to avoid the situation last week with the unnoticed failures for the command-line utils.

[cwhitlock-NOAA]

After chatting with Ian on 2024-07-15, 1 is currently covered by the conda build tests thanks to the meta.yaml file - but the meta.yaml produces one big logfile that's hard to parse through. Thus, it's a good idea to separate those tests into 3 components:

pylint command: pylint fre/ runs: on push reason: the more often we linter the code, the easier it is to keep edits in line

pytest command:

• pytest fre/tests/
• pytest fre/app/generate_time_averages/tests runs: on push reason: It's possible to want to push code to the repo before it runs properly. However, that code should not get into the main branch, where someone other than us may run it.

did-it-build tests command: --help of all click tools/subtools runs: on push reason: see pytest

• pytest and did-it-build tests both would benefit from a "call without invoking ci" option (test script?)
• all tests should run in the built instance of fre-cli from the ci/cd tests if invoked during ci

[ilaflott]

did-it-build tests command: --help of all click tools/subtools runs: on push reason: see pytest

this should be part of the pytest suite we write- not it's own category. the does it build test is the conda build . line in the github workflow, build_conda.yml.

fre --help, fre <tool> --help, and fre <tool> <subtool> --help are all calls that can/should be tested in pytest, using click's CliRunner class. see fre/tests for some barebones examples.

[cwhitlock-NOAA]

Thank you for the clarification - I had not been grouping the explicit invocations of pytest in the same category as the invocations of what are currently command-line tests.

Second philosophical question: should we be listing out each of the subtools and hard-coding them in the test itself, or trying to determine them programmatically like this:

for each $subtool in $subtools: runner = CliRunner() result = inner.invoke(fre, ["--help", $subtool,"pp"]) assert result.exit_code == 0

On Mon, Jul 15, 2024 at 1:21 PM Ian @.***> wrote:

did-it-build tests command: --help of all click tools/subtools runs: on push reason: see pytest

this should be part of the pytest suite we write- not it's own category. the does it build test is the conda build . line in the github workflow, build_conda.yml https://github.com/NOAA-GFDL/fre-cli/blob/main/.github/workflows/build_conda.yml .

fre --help, fre --help, and fre --help are all calls that can/should be tested in pytest, using click's CliRunner class. see fre/tests https://github.com/NOAA-GFDL/fre-cli/tree/main/fre/tests for some barebones examples.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-GFDL/fre-cli/issues/122#issuecomment-2229014521, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC362WUHEHKQNFASFSBM4WLZMQAHTAVCNFSM6AAAAABK4TTHGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRZGAYTINJSGE . You are receiving this because you were assigned.Message ID: @.***>

-- Carolyn Whitlock

[ilaflott]

We should avoid snazzy machinery that dynamically-finds subtools or certain pieces of code. The tests should be written as close to the one-line-one-test philosophy as we can muster.

In terms of repo structure + tests- we currently have:

fre/tests/ # --> this tests `fre.py`, and e.g. `fre --help` or just `fre`
fre/app/generate_time_averages/tests # this only tests `fre app generate-time-averages` type things

so i would think that therefore: fre/app/tests/ holds tests for e.g. fre app, fre app --help.

... fre/make/tests holds tests for fre make etc. So when we do this for everything, if we just do pytest fre/, it should find and list all those fre/.../tests directories + the test scripts pytest is running

[ilaflott]

Thank you for the clarification - I had not been grouping the explicit invocations of pytest in the same category as the invocations of what are currently command-line tests. Second philosophical question: should we be listing out each of the subtools and hard-coding them in the test itself, or trying to determine them programmatically like this: for each $subtool in $subtools: runner = CliRunner() result = inner.invoke(fre, ["--help", $subtool,"pp"]) assert result.exit_code == 0 … On Mon, Jul 15, 2024 at 1:21 PM Ian @.> wrote: did-it-build tests command: --help of all click tools/subtools runs: on push reason: see pytest this should be part of the pytest suite we write- not it's own category. the does it build test is the conda build . line in the github workflow, build_conda.yml https://github.com/NOAA-GFDL/fre-cli/blob/main/.github/workflows/build_conda.yml . fre --help, fre --help, and fre --help are all calls that can/should be tested in pytest, using click's CliRunner class. see fre/tests https://github.com/NOAA-GFDL/fre-cli/tree/main/fre/tests for some barebones examples. — Reply to this email directly, view it on GitHub <#122 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC362WUHEHKQNFASFSBM4WLZMQAHTAVCNFSM6AAAAABK4TTHGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRZGAYTINJSGE . You are receiving this because you were assigned.Message ID: @.> -- Carolyn Whitlock

I was looking for something else today and stumbled on this- seems relevant. pytest.ini for pytest configuration, and we can specify specific test-targets there.