Harnesses for distribution tests

hkershaw-brown commented 2 months ago

Description:

Adds developer tests for normal_dist beta_dist gamma_dist

from https://github.com/NCAR/DART/commit/5b3850fb1a08fff0ebce3f95283fef29de431bdc

note no bnrh test since there is no test_bnrh in bnrh_distribution_mod.f90

note the pass/fail conditions for the matlab tests do no match the comments for gamma and beta. The comment says absolute value, the test is not on the absolute value.

Fixes issue

This pull request does not fix any of the failing tests.

See #717 for discussion of beta failing tests. Normal distribution has failed to converge on ifort (IFORT) 2021.10.0 20230609 Derecho

hkershaw@derecho3:/glade/derecho/scratch/hkershaw/DART/stats-tests/developer_tests/normal_dist/work(stats-tests)$ ./test_normal_dist  

 --------------------------------------
 Starting ... at YYYY MM DD HH MM SS = 
                 2024  9  3 12 16 34
 --------------------------------------

  set_nml_output Echo NML values to log file only
 Matlab Comparison Tests: PASS   3.053113317719180E-016
  inv_cdf  Failed to converge for quantile   0.999993136435131
  inv_cdf  Failed to converge for quantile   0.999993263468024
  inv_cdf  Failed to converge for quantile   0.999999082322113
  inv_cdf  Failed to converge for quantile   0.999999852396711
  inv_cdf  Failed to converge for quantile   0.999999883735506
  inv_cdf  Failed to converge for quantile   0.999999927199702
  inv_cdf  Failed to converge for quantile   0.999999927201206
  inv_cdf  Failed to converge for quantile   0.999999927202710
  inv_cdf  Failed to converge for quantile   0.999999927204214
  inv_cdf  Failed to converge for quantile   0.999999927211732
  inv_cdf  Failed to converge for quantile   0.999999927213236
  inv_cdf  Failed to converge for quantile   0.999999927214740
  inv_cdf  Failed to converge for quantile   0.999999927220754
  inv_cdf  Failed to converge for quantile   0.999999947103601
  inv_cdf  Failed to converge for quantile   0.999999947108017
  inv_cdf  Failed to converge for quantile   0.999999961505014
  inv_cdf  Failed to converge for quantile   0.999999961508261
  inv_cdf  Failed to converge for quantile   0.999999961517999
  inv_cdf  Failed to converge for quantile   0.999999961520434
  inv_cdf  Failed to converge for quantile   0.999999961521245
  inv_cdf  Failed to converge for quantile   0.999999961525302
  inv_cdf  Failed to converge for quantile   0.999999961527736
  inv_cdf  Failed to converge for quantile   0.999999961529358
  inv_cdf  Failed to converge for quantile   0.999999961531792
  inv_cdf  Failed to converge for quantile   0.999999961532603
  inv_cdf  Failed to converge for quantile   0.999999961533414
  inv_cdf  Failed to converge for quantile   0.999999961534225
  inv_cdf  Failed to converge for quantile   0.999999961535036
  inv_cdf  Failed to converge for quantile   0.999999961535847
  inv_cdf  Failed to converge for quantile   0.999999966830631
  inv_cdf  Failed to converge for quantile   0.999999966835549
  inv_cdf  Failed to converge for quantile   0.999999983976664
  inv_cdf  Failed to converge for quantile   0.999999983977011
  inv_cdf  Failed to converge for quantile   0.999999983980134
  inv_cdf  Failed to converge for quantile   0.999999983981522
  inv_cdf  Failed to converge for quantile   0.999999983982563
  inv_cdf  Failed to converge for quantile   0.999999983984645
  inv_cdf  Failed to converge for quantile   0.999999983985339
  inv_cdf  Failed to converge for quantile   0.999999983986726
  inv_cdf  Failed to converge for quantile   0.999999983991235
  inv_cdf  Failed to converge for quantile   0.999999983992622
  inv_cdf  Failed to converge for quantile   0.999999983997476
  inv_cdf  Failed to converge for quantile   0.999999983997823
 PASS: Max inversion diff   3.552713678800501E-015  < bound 
...

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation update

Documentation changes needed?

[ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.

Tests

Please describe any tests you ran to verify your changes.

Checklist for merging

[ ] Updated changelog entry
[ ] Documentation updated
[ ] Update conf.py

Checklist for release

[ ] Merge into main
[ ] Create release from the main branch with appropriate tag
[ ] Delete feature-branch

Testing Datasets

[ ] Dataset needed for testing available upon request
[ ] Dataset download instructions included
[x] No dataset needed

mjs2369 commented 1 month ago

The code is good, allows you to easily use the test_x subroutines in the various distribution modules and makes it more obvious when the checks fail.

note the pass/fail conditions for the matlab tests do no match the comments for gamma and beta. The comment says absolute value, the test is not on the absolute value.

With regards to this comment above that you made in the PR, I think the comments do match the code. The comments say that the "Absolute value of differences should be less than 1e-15", and the conditionals are if(abs(maxval(pdf_diff)) < 1e-15_r8 .and. abs(maxval(cdf_diff)) < 1e-15_r8) which is using the absoulte value function.

I'm seeing the same results with regards to getting a bunch of "inv_cdf Failed to converge for quantile" messages for normal_distribution, but I am using gfortran instead of Intel.

I'm also getting one of these messages from inv_cdf in the results of beta_test: inv_cdf Failed to converge for quantile 0.0000000000000000 I can't really tell if this problematic or not for a quantile of 0, but I think we are probably good to go ahead and merge this once I get your thoughts on this @hkershaw-brown

hkershaw-brown commented 1 month ago

Hi Marlee, thanks for looking at this, I'm leaving this pull request hanging out at the moment. It was an attempt to break up this mega commit from Jeff: https://github.com/NCAR/DART/commit/5b3850fb1a08fff0ebce3f95283fef29de431bdc

There are some problems with the QCEFF, so I don't want to dump these tests into a release at the moment.

FYI "the comments do match the code" because I changed the code. Anyway sorry this is such a round-about way of developing.

NCAR / DART