[JOSS Review] Testing

[CFGrote]

Re JOSS #4509

Testing

I could not find instructions for how to run tests. I found that running

$> pytest -v

in the root dir works, though.

tests/test_is_simulation.py::TestIS_Simulation::test_is_simulation generates two matplolib plot windows that need to be closed manually before the tests continue. I find that slightly annoying as tests cannot be run automatically. As I'd like to recommended adding the tests to a CI setup (e.g. github-actions), this would have to be addressed. One solution could be to check if the test is run interactively or within a CI automated setup and have the test generate these plots or not.

I suggest adding information about how to run tests to the README and documentation.

Also, there seems to be some coverage testing functionality as indicated by lines 16,17 in setup.cfg, would be great to learn how to run coverage testing and get a coverage report. I assume coverage is great (looking at the impressive number of test modules), but I'd like to see the actual numbers ;)

[Jacks0nJ]

I've now added instructions for testing the code in the README.

The two plots being being generated during testing has now been suppressed. The code already has CI setup; the plots appearing are a rare event to alert the user there could be a problem, so simply by chance had not been generated during previous testing of the code using unit tests.

As for coverage testing, I've struggled to implement it. Do you have a recommended method?

(The problem has been I either need to use a third-party provided, which I don't really want to do, or the locally done coverage tests simply break. It seems to be to do with how I set up testing; locally run tests work on the installed package (rather than local code), such that it tests the uploaded code when I push to Github (this is becuase it needs to compile Cython). But it appears coverage.py (and similair packages) can't find the functions to be tested, as it is not in the directory, e.g. test_is_simulation can't find pyfpt.numerics.is_simulation when I run coverage tests.)

[CFGrote]

Hi @Jacks0nJ, I'm currently on vacation with very limited access to internet. i'll take this up again when i'm back on august 19th. hope that does not cause too much trouble on your side.

best, Carsten

[Jacks0nJ]

Hi @CFGrote That's absolutely no issue! Enjoy your vacation!

Best, Joe

[CFGrote]

Ok, I'm back :sun_with_face::tropical_drink:

What works for me is:

coverage run -m pytest tests/

to run the coverage test, followed by

coverage report

to generate the report:

Name                                                    Stmts   Miss  Cover
---------------------------------------------------------------------------
pyfpt/__init__.py                                           2      0   100%
pyfpt/analytics/__init__.py                                18      0   100%
pyfpt/analytics/classicality_criterion.py                  12      0   100%
pyfpt/analytics/edgeworth_pdf.py                           28      0   100%
pyfpt/analytics/fourth_central_moment_efolds.py            19      0   100%
pyfpt/analytics/gaussian_deviation.py                      31      1    97%
pyfpt/analytics/gaussian_pdf.py                            12      0   100%
pyfpt/analytics/kurtosis_efolds.py                         10      1    90%
pyfpt/analytics/mean_efolds.py                             20      0   100%
pyfpt/analytics/optimal_bias_amplitude.py                  38     17    55%
pyfpt/analytics/quadratic_inflation_large_mass_pdf.py      45     37    18%
pyfpt/analytics/quadratic_inflation_near_tail_pdf.py       28     23    18%
pyfpt/analytics/reduced_potential.py                        8      0   100%
pyfpt/analytics/reduced_potential_ddiff.py                  8      0   100%
pyfpt/analytics/reduced_potential_diff.py                   8      0   100%
pyfpt/analytics/skewness_efolds.py                         22      0   100%
pyfpt/analytics/slow_roll_diffusion.py                      6      0   100%
pyfpt/analytics/slow_roll_drift.py                          4      0   100%
pyfpt/analytics/third_central_moment_efolds.py             20      0   100%
pyfpt/analytics/variance_efolds.py                         20      0   100%
pyfpt/numerics/__init__.py                                 12      0   100%
pyfpt/numerics/data_in_histogram_bins.py                   19      0   100%
pyfpt/numerics/data_points_pdf.py                          54      4    93%
pyfpt/numerics/histogram_data_truncation.py                31      1    97%
pyfpt/numerics/histogram_normalisation.py                   6      1    83%
pyfpt/numerics/is_simulation.py                            82     32    61%
pyfpt/numerics/jackknife_errors.py                         31      4    87%
pyfpt/numerics/log_normal_error.py                         10      0   100%
pyfpt/numerics/log_normal_height.py                         4      0   100%
pyfpt/numerics/log_normal_mean.py                           7      0   100%
pyfpt/numerics/lognormality_check.py                       42     18    57%
pyfpt/numerics/multi_processing_error.py                    8      0   100%
pyfpt/numerics/save_data_to_file.py                        12      1    92%
tests/test_classicality_criterion.py                       22      1    95%
tests/test_data_in_histogram_bins.py                       16      1    94%
tests/test_data_points_pdf.py                              28      1    96%
tests/test_edgeworth_pdf.py                                27      1    96%
tests/test_fourth_central_moment_efolds.py                 23      1    96%
tests/test_gaussian_deviation.py                           23      1    96%
tests/test_gaussian_pdf.py                                 27      1    96%
tests/test_histogram_data_truncation.py                    25      1    96%
tests/test_histogram_normalisation.py                      18      1    94%
tests/test_importance_sampling_cython.py                   56      1    98%
tests/test_is_simulation.py                                39      1    97%
tests/test_jackknife_errors.py                             24      1    96%
tests/test_kurtosis_efolds.py                              23      1    96%
tests/test_log_normal_error.py                             14      1    93%
tests/test_log_normal_height.py                            14      1    93%
tests/test_log_normal_mean.py                              13      1    92%
tests/test_lognormality_check.py                           35      1    97%
tests/test_mean_efolds.py                                  23      1    96%
tests/test_multi_processing_error.py                       17      1    94%
tests/test_optimal_bias_amplitude.py                       24      1    96%
tests/test_reduced_potential.py                            14      1    93%
tests/test_reduced_potential_ddiff.py                      14      1    93%
tests/test_reduced_potential_diff.py                       14      1    93%
tests/test_save_data_to_file.py                            29      1    97%
tests/test_skewness_efolds.py                              23      1    96%
tests/test_slow_roll_diffusion.py                          28      5    82%
tests/test_slow_roll_drift.py                              26      3    88%
tests/test_third_central_moment_efolds.py                  23      1    96%
tests/test_variance_efolds.py                              23      1    96%
---------------------------------------------------------------------------
TOTAL                                                    1362    175    87%

Not bad, actually :smiley:

But maybe I don't understand your exact problem?

[Jacks0nJ]

Great, hope you had a good holiday!

And I'm glad the unit tests are working!

Sorry, I thought you were originally suggesting having a coverage badge on the README which automatically displays the coverage. But this was proving very difficult and your method is a lot easier.

[CFGrote]

Great, hope you had a good holiday! Totally, although a bit short, as always.

And I'm glad the unit tests are working! Yep

Sorry, I thought you were originally suggesting having a coverage badge on the README which automatically displays the coverage. But this was proving very difficult and your method is a lot easier. No, that was not my intention. I think we can close this one now.