Regression testing

[SouthEndMusic]

There are a few use cases for testing which are quite expensive:

• Run models with various solvers and compare their results;
• Run models with a sparse Jacobian (https://github.com/Deltares/Ribasim/pull/459) and a dense one and compare the results.

These could possibly also involve large models like the one mentioned in https://github.com/Deltares/Ribasim/issues/419#issue-1799111085.

@evetion suggested regression testing for this.

[evetion]

So we want:

• [ ] Storage of output values
• [ ] Comparison of output values (~)
• [ ] Running this on TeamCity every week or so
• [ ] Running specific Julia tests only on TC

Open for discussion, is this a Python tool, or do we do this in pure Julia? Probably should follow the same ideas as in #151, part of #419.

[evetion]

Check in with the coupler, as they already do some reference checks.

[Jingru923]

Regression test in Ribasim should have following content which I think it's most important for now. I will build benchmark for these content.

I selected 4 representative small test models: basic, pid_control, trivial, subnetwork_with_sources. With 7 implicit different solvers, Jacobian densities, autodiff on-off, the following output value will be check:

• The start and the end of basin storages
• Basin storage convergences
• The start and the end of basin levels
• Basin level convergences
• balance_error of basins
• relative_error of basins
• any idea for subgrid_leve?

For the larger models, their performance is also important. I see overlapping with issue #1471. So they will be worked on in coming sprint.

[Jingru923]

When testing different solver for trivial model (a simple model) all solver can give very approximate result (results of different solver can ≈ each other).

However, this is not the case for the more complex model basic and pid_control. I believe same will happen for the subnetwork_with_sources model. Due to the time limit, I will search for a good margin for all the different solvers to pass the regression tests. For long-term goal, we should revise if all the solvers are actually suitable and necessary for Ribasim, and which one is better to keep