Consider this a 'work in progress' issue. It should serve as starting point for further discussion on the topic and its content is expected to change in the future.
Motivation
Over time, the testing scripts and overall testing procedure has become quite muddled and intransparent. As a result, the process of running systemtests is both hard to understand and cumbersome to manage. With the transition from TravisCI to Github Actions coming (hopefully) soon, sweeping changes to the codebase are expected and would be a good opportunity to also start thinking about how to improve the testing procedure overall.
Current problems that are intrinsic to the testing structure
The naming scheme for docker images is not complete and is not clearly defined.
We currently determine the name of an image by combining the following characteristics:
Type of the image (preCICE build, adapter build, solver build)
Base image used (e.g. Ubuntu 16.04/18.04/20.04) and installation method used (.home/.package)
Branch of the preCICE repository that the build was made from.
Optional features. These include petsc for preCICE built with PETSc, mpich for preCICE built with MPICH instead of OpenMPI.
Note that naming is done within several modules independently from each other, meaning that changes to the features need to be applied at multiple locations. Additionally, the current scheme does not take into account:
the branch of the adapter repository
the exact revision of a branch.
Firstly, it would be good to have this naming scheme be defined in one central location of the code, from which the different modules then draw to either assemble a name for an image or extract features from a given name.
The comparison script is not used to its fullest potential.
(explanation to follow)
The module used for triggering systemtests from other repositories is disconnected from the main testing structure.
(explanation to follow)
Changes to reference data are cumbersome to implement.
(explanation to follow)
There exists no structure for building solver images automatically.
Consider this a 'work in progress' issue. It should serve as starting point for further discussion on the topic and its content is expected to change in the future.
Motivation
Over time, the testing scripts and overall testing procedure has become quite muddled and intransparent. As a result, the process of running systemtests is both hard to understand and cumbersome to manage. With the transition from TravisCI to Github Actions coming (hopefully) soon, sweeping changes to the codebase are expected and would be a good opportunity to also start thinking about how to improve the testing procedure overall.
Current problems that are intrinsic to the testing structure
The naming scheme for docker images is not complete and is not clearly defined.
We currently determine the name of an image by combining the following characteristics:
.home/.package)petscfor preCICE built with PETSc,mpichfor preCICE built with MPICH instead of OpenMPI.Note that naming is done within several modules independently from each other, meaning that changes to the features need to be applied at multiple locations. Additionally, the current scheme does not take into account:
Firstly, it would be good to have this naming scheme be defined in one central location of the code, from which the different modules then draw to either assemble a name for an image or extract features from a given name.
The comparison script is not used to its fullest potential.
(explanation to follow)
The module used for triggering systemtests from other repositories is disconnected from the main testing structure.
(explanation to follow)
Changes to reference data are cumbersome to implement.
(explanation to follow)
There exists no structure for building solver images automatically.
(explanation to follow)