Model test

[TheVincentWagner]

Ideally, a new model could be tested for eulerpi-compatibility with a single function call.

[kaiserls]

Hey, I think an end-to-end test for the applicability of models is very useful. The test in its current form could however be hard to evaluate due to run time restrictions, fail at an early stage due to implementation errors in the model, or the plot results hard to interpret. Could we maybe divide/extend the model check?

1. Catch implementation errors (e.g. declared data dim does not match model output dim)
2. Catch model errors (model jacobian does not have full rank for central param)
3. Catch other remaining problems preventing eulerpi from successful inference.

Step 1 would be very cheap to run (~1 model evaluation) and is already implemented in tests/test_examples.py:test_model_requirements. Step 2 is a bit more expensive (~3 model evaluations) and could become more expensive when we want e.g. also check the jacobian at the param limits. It is partially implemented at tests/test_examples.py:test_model_requirements. Step 3: Can you explain what problems could prevent a successful application of eulerpi at this stage? We could include this in the documentation then at a later point.

Step 1 and 2 could also be executed at the beginning of the inference function when set via a parameter check_model=True

PS: I just saw your comment on issue #41: I would mostly divide between testing our library and testing external user-defined models. Of course, we can test our own example models using the same code with which we can check external user-defined models.

[TheVincentWagner]

Hi Lars,

I very much like your idea. Do I understand correctly that you mainly propose to divide the currently single check into several and adding more try/catch blocks? I also like the idea of having an (optional) model check before inference.

Concerning issue #41: I did not mean to touch this issue with this current issue. I just thought a further extension of our internal test suite would not hurt.

By the way: Do yuóu understand why CI currently fails?

[kaiserls]

I propose to offer the user not only the full, expensive end-to-end check that eventually needs interpretation but also the checks, which we do for our own models internally. See tests/test_examples.py:test_model_requirements. I don't know if assert ... or if ... raise ...Exception is better. With try-catch, we could catch errors that are raised internally in the users model. We could reraise it and add the information that the error is in their model, but this should already be apparent when the user inspects the error message, because Python always has a stack trace, at least for Python code.

CI fails because the file test_models.py (or how exactly it is named) still exists and is found by the test suit, I think, and trying to execute the function model_test without passing arguments. You can delete the old file and rename the function inside your new file to full_model_check or something like that. This would make the check_model model_check less confusing.

PS: Could/should/can we also give the user some indication of whether the results achieved with the full check are good? Just like we do it with the convergence test in our test suit again

[TheVincentWagner]

Hi Lars,

I indeed used your implementation from the test suite to create a basic checking functionality. In addition to that, the initial check I wrote is now the inference check of the model. You can either use both functions individually or call the full_model_test to call them in succession. This last option is our recommendation for a new model. All checks are accompanied by hopefully informative print statements that can clarify what goes wrong at what point. Within the test_examples function, we now call the basic_model_check to avoid code doubling. Are you ok with this solution?

Thank you for helping me with the CI. It should run now.

I am currently working on implementing the Kolmogorov-Smirnov test between data and data reconstruction as a quantitative assessment of the inference quality, but I will submit this in a separate issue as the statistics are a bit tricky and need thorough treatment.