search

OasisLMF / ktools

In-memory simulation kernel for loss modelling.
BSD 3-Clause "New" or "Revised" License
28 stars 19 forks source link

Potential for missing files during testing to go unnoticed #299

Closed hchagani-oasislmf closed 2 years ago

hchagani-oasislmf commented 2 years ago

Issue Description

Since the introduction of parquet output in ktools v3.8.0, the build script searches for parquet libraries and links to them during compilation if found. Parquet output is not supported on Mac OS builds, and therefore problems were found with missing files during testing (see issue https://github.com/OasisLMF/ktools/issues/285). The solution (see PR https://github.com/OasisLMF/ktools/pull/288) was to ignore checksums for missing files by adding the --ignore-missing argument. In other words, if parquet files were not produced as the parquet libraries were not present during compilation, their checksums would be ignored and no error would be returned.

However, this will not only ignore missing parquet files, but has the potential to ignore missing csv files that should be present. For example, if a csv file is expected from aalcalc, but there is an issue and no file is produced, the tests will erroneously pass.

One possible solution is to separate checksums from parquet and non-parquet files. If the parquet libraries have been linked during compilation, affected ktools components such as katparquet will include Parquet output enabled in their output when the version number is queried. For example:

$ katparquet -v
./katparquet : version: 3.8.0 - git update: Wed May 11 11:49:26 2022 +0100t : Parquet output enabled

Exploiting this, the following would only run tests on the parquet files if the parquet libraries were linked to during compilation:

if ../src/katparquet/katparquet -v 2>&1 | grep -q "Parquet output enabled"
then
    shasum -c ../$CTRL_PARQUET.sha1
fi

where $CTRL_PARQUET.sha1 contains checksums for the parquet files.

Version / Environment information

ktools v3.8.1, as this is when missing files are ignored. In the previous version, ktools v3.8.0, the parquet output was not tested.

hchagani-oasislmf commented 2 years ago

For future reference, separate files containing checksums can be easily created using the following from the testout/ directory:

$ sha1sum !(*.parquet) > ../ctrl.sha1
$ sha1sum *.parquet > ../ctrl_parquet.sha1