search

NOAA-FIMS / FIMS-planning

:fish: :chart_with_upwards_trend: :ocean: a place to manage FIMS' planning process
2 stars 0 forks source link

Investigation group idea: testing #18

Closed Bai-Li-NOAA closed 2 years ago

Bai-Li-NOAA commented 3 years ago

Some thoughts on test-driven development based on the r4MAS testing work. It would be great to create unit tests, functional tests, and integrated tests to introduce as many automated checks as possible to validate models before the models are fully developed. Once those test cases are created, they can be added to a continuous integration tool (e.g. GitHub Action) so that every time a commit or a pull request is merged in on GitHub, this suite of tests is run. It may have overlap with the collaborative workflow issue https://github.com/NOAA-FIMS/FIMS-planning/issues/17 if we need to create testing procedure template.

This fishsad - Fisheries Stock Assessment Data repository is used to store assessment input and output data for cross-model testing. The assessment data repository can be linked with another repository, which contains functions that facilitate the conversion of stock assessment input files from one framework to another (e.g. saconvert). Would be curious to hear opinions from others on setting up FIMS testing framework.

Rick-Methot-NOAA commented 3 years ago

I encourage feedback on this topic from @k-doering-NOAA as she has set up a highly functional test system using Jenkins for Stock Synthesis

k-doering-NOAA commented 3 years ago

@Rick-Methot-NOAA , luckily I asked to join this group with Bai already!

I added @Bai-Li-NOAA and myself to assignees to indicate we are planning on participating in this group (not sure if there is a better way to do this). Feel free to assign yourself, too if anyone else wanted to join this investigation group.

Bai-Li-NOAA commented 3 years ago

Scope This group will focus on investigating testing frameworks to be applied for FIMS. We aim to discuss how to compare FIMS model runs with previous FIMS model versions, other modeling platforms, and true values from operating models through various tests. We plan to discuss error checks for FIMS input and output (i.e., run-time testing) but will check with the input/output file and type structure group first to avoid overlap. More specifically, we will:

1) share testing experience and review test cases from various platforms (stock assessment platforms and other software projects). We can compile a list of examples and discuss how to set up a testing framework structure, how often to run each type of test, and how to automate acceptance tests. For automated acceptance tests, we can discuss how to create expectation and evaluation criteria (e.g., tolerance value).

2) discuss various types of tests that could be useful for FIMS development. For example, why, who, when, and how to conduct unit tests, functional tests, integrated tests, regression tests, compatibility tests, and one-off tests (e.g., for replicating and fixing user-reported bugs). We plan to provide a glossary of terms and simple examples for each type of test. We can work with @Andrea-Havron-NOAA and see how tests can be implemented across different software languages under consideration for FIMS.

3) discuss FIMS test procedures. For example, draft a test procedure template and discuss the steps for conducting individual tests.