Add new Ensemble-Stat line type to store the decomposition of the empirical CRPS into reliability and potential.

JohnHalleyGotway commented 3 years ago

Describe the New Feature

MET #1450 adds the empirical CRPS and corresponding skill score to the ECNT line type, as described in Hersbach, 2000. During that development, the UK MetOffice pointed out that their existing verification system (ver) is able to decompose the CRPS into its components: CRPS = Reliability + Potential. Doing so for a single case is straight-forward enough but aggregating those scores over multiple cases is challenging. Properly doing so requires that an alpha and beta value be stored for each ensemble member. Adding 2 * number of ensemble members to the existing ECNT line type was not practical, and so the decomposition was not computed.

This task is to add a new Ensemble-Stat line type for the CRPS decomposition. It should include the CRPS_EMP, CRPS_REL, CRPS_UNC, and the alpha and beta values for each member. Also for this issue, ensure that Stat-Analysis can aggregate multiple lines together.

Don't forget to define corresponding METdatadb and/or METviewer issues to store this new line type!

Acceptance Testing

List input data types and sources. Describe tests required for new functionality.

Time Estimate

Estimate the amount of work required here. Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.

[ ] Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

[ ] Select engineer(s) or no engineer required
[ ] Select scientist(s) or no scientist required

Labels

[x] Select component(s)
[x] Select priority
[x] Select requestor(s)

Projects and Milestone

[ ] Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
[ ] Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

[x] METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy
NEED TO DEFINE METVIEWER ISSUE TO LOAD/PLOT THIS NEW LINE TYPE.
NEED TO DEFINE METDATADB ISSUE TO LOAD THIS NEW LINE TYPE.

New Feature Checklist

See the METplus Workflow for details.

[ ] Complete the issue definition above, including the Time Estimate and Funding source.
[ ] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
[ ] Complete the development and test your changes.
[ ] Add/update log messages for easier debugging.
[ ] Add/update unit tests.
[ ] Add/update documentation.
[ ] Push local changes to GitHub.
[ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
[ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s), Project(s), Milestone, and Linked issues
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Close this issue.

JohnHalleyGotway commented 3 years ago

Here are some development notes I used when testing the python and R packages for doing the empirical CRPS decomposition: Dev_Notes.txt

JohnHalleyGotway commented 2 years ago

@mpm-meto, is this feature required for MET-11.0.0-beta4 or can it be pushed to the next release in favor of other higher priority issues?

dtcenter / MET