Quantile methods

[hkershaw-brown]

Description:

Adds a Quantile-Conserving Ensemble Filtering Framework to DART Publications: QCEFF part1 QCEFF part 2.

The default QCEFF options are EAKF, normal distribution (no bounds).

This is a breaking change and will be v11 of DART Namelist item removed from filter_nml

• filter_kind - now a per qty option

Two new required namelists (added to input.nml files):

• &probit_transform_nml
• &algorithm_info_nml

assim_tools_mod

• sort_obs_inc namelist option applied to ENKF only, so default is now .true.
• spread_restoration is not supported in this version

algorithm_info_mod

• runtime QCEFF options read from .csv or .txt file

New probability distribution modules:

• beta_distribution_mod

• bnrh_distribution_mod (bounded normal rank histogram)

• gamma_distribution_mod

• normal_distribution_mod

• probit_transform_mod

• distribution_params_mod

Update to lorenz_96_tracer_advection:

• positive_tracer
• default mean velocity = 0)
• perturbation
• real(r8) not real(i8) see #615

1d forward operator updates:

• For non-integer powers, fix up values for negative bases

Docs:

• index.rst: Nonlinear and Non-Gaussian Data Assimilation Capabilities in DART
• QCEFF instructions: Quantile-Conserving Ensemble Filter Framework
• Example to work through: QCEFF: Examples with the Lorenz 96 Tracer Model

Fixes issue

review notes #458 Note this pull request includes the fix for #416

Todo:

• [ ] continuing existing experiments #477
• [x] Remove models/lorenz_96/work/README (readme file for papers)
• [x] remove qcf warning before merging into main: qceff_probit.rst, qceff-examples.rst, index.rst
• [x] change log add contributors (Chris Riedel)
• [ ] After merge, create 'quantile_methods' tag (with a warning) so old readthedocs links point to docs. No, edit release notes not to point at quantile_methods.
• [ ] change create_obs_sequence input file for lorenz_96 tracer?
• [x] developer_test from reading table algorithm_info_nml

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] Documentation update

Documentation changes needed?

• [x] My change requires a change to the documentation.
  • [x] I have updated the documentation accordingly.

Tests

Please describe any tests you ran to verify your changes. bitwise lorenz_96 with main for default qcf options Jeff has run many algorithm tests. Speed tests main vs. default qcf options Derecho

Timing:

• main vs. qceff default Derecho
• bhrf vs main Cheyenne

Checklist for merging

• [x] Updated changelog entry ⚠️ ⚠️ ⚠️ ⚠️ 🧇
• [x] Documentation updated (remove WARNING)
  • [x] qceff_probit.rst remove 'alpha release stage' ⚠️
  • [x] qceff-examples.rst remove 'alpha release stage' ⚠️
  • [x] index.rst remove WARNING ⚠️
• [x] Update conf.py

Checklist for release

• [x] Merge into main
• [x] Create release from the main branch with appropriate tag
• [x] Delete feature-branch

Testing Datasets

• [x] Dataset needed for testing available upon request
• [ ] Dataset download instructions included
• [x] No dataset needed

[hkershaw-brown]

note to reviewers, documentation for this pull request is here: https://dart-documentation--588.org.readthedocs.build/en/588/

[hkershaw-brown]

@jlaucar can you go ahead and add whoever you want to the list of reviewers. Cheers, Helen

[mjs2369]

These comments are now incorrect and should be removed since we are once again dividing by sqrt(2 PI)

https://github.com/NCAR/DART/blob/d5c21646ebef17865cfe779fd68894cbe6a51112/assimilation_code/modules/assimilation/assim_tools_mod.f90#L1701C1-L1702C48

https://github.com/NCAR/DART/blob/d5c21646ebef17865cfe779fd68894cbe6a51112/assimilation_code/modules/assimilation/assim_tools_mod.f90#L1714C1-L1715C48

[hkershaw-brown]

I fixed the typos mentioned in the comments of this review in PR six twelve. Approving the six twelve merge will reconcile the typos and will add substantive edits to index.rst.

I added the substantive edits to index.rst because the quantile methods merge adds a lot of new information to the documentation that DART users may be unfamiliar with. It may be easier to read by following a standard practice in technical writing: using topic and stress positions in sentences. The topic position in a sentence contains information that a user already knows, or is presumed to know, while the stress position introduces new information.

doc changes merge in from #612 pull request.

[hkershaw-brown]

The &probit_transform_nml and &algorithm_info_nml are missing from models/MITgcm_ocean/work/input.nml

Add these in 8ed6c2e

Add for any model/input.nml that was missing these.

#!/bin/bash
files=$(find . -name input.nml)

for f in $files
do
   if ! grep -q probit_transform_nml  $f; then
     echo probit not found $f
   fi

done

for f in $files
do
  if ! grep -q algorithm_info_nml  $f; then
     echo algorithm not found $f
   fi  
done