Review Computation of No-Skill reference line on Reliability Plots

TaraJensen commented 3 years ago

Describe the Problem

See METcalcpy #26

Expected Behavior

See METcalcpy #26

Environment

See METcalcpy #26

To Reproduce

Describe the steps to reproduce the behavior: 1. Go to '...' 2. Click on '....' 3. Scroll down to '....' 4. See error Post relevant sample data following these instructions: https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp

Relevant Deadlines

This is a bug and has high priority - assigning to beta1 because there is not bugfix project right now

Funding Source

See METcalcpy #26

Define the Metadata

Assignee

[ ] Select engineer(s) or no engineer required
[ ] Select scientist(s) or no scientist required

Labels

[ ] Select component(s)
[ ] Select priority
[ ] Select requestor(s)

Projects and Milestone

[ ] Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
[ ] Select milestone to relevant bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

[ ] METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Bugfix Checklist

See the METplus Workflow for details.

[ ] Complete the issue definition above, including the Time Estimate and Funding Source.
[ ] Fork this repository or create a branch of main_\. Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
[ ] Fix the bug and test your changes.
[ ] Add/update log messages for easier debugging.
[ ] Add/update unit tests.
[ ] Add/update documentation.
[ ] Push local changes to GitHub.
[ ] Submit a pull request to merge into main_\. Pull request: bugfix <Issue Number> main_<Version> <Description>
[ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s), Project(s), Milestone, and Linked issues
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Complete the steps above to fix the bug on the develop branch. Branch name: bugfix_<Issue Number>_develop_<Description> Pull request: bugfix <Issue Number> develop <Description>
[ ] Close this issue.

TatianaBurek commented 3 years ago

Possible solution: For the no-skill line to have a slope of 0.5 (half the slope of the perfect-reliability line) and it should have intercept 0.5 * env$o_bar[??] (i.e., half the value of the no-resolution) change:

a=env$dblNoSkillB[i][[1]], # intercept
 b=env$dblNoSkillM[i][[1]], # slope

to

a=0.5*env$o_bar[i][[1]], # intercept
b=0.5, # slope

TatianaBurek commented 3 years ago

There should also be a line with abline( v = env$o_bar[i][[1]] ) Eric has not found a name for this line, but all the attributes diagrams in all the literature have it. Typically, these plots shade the areas in the box in the lower left corner made by the horizontal and vertical lines and between the no-skill line and the vertical line in the upper right corner. Eric thinks polygon will do the shading in R, but if not, he can try to find it again. Basically, everything inside this shading represents a "good" forecast.

TatianaBurek commented 3 years ago

Eric suggested to include all lines descriptions including the no-skill line to the legend. Need to discuss with users

TatianaBurek commented 3 years ago

from Eric: The solution I came up with based on Hsu and Murphy's paper is to make a line with slope of 0.5 and intercept equal to half the no-resolution value, which gives you a line equidistant between the perfect reliability (slope = 1) and no-resolution (slope = 0) lines. When doing so, I get a line that is exactly where it should be according to every other example I've seen.

I made a few additional suggestions to add to the diagram based on most other such diagrams I've seen. Namely, to add a vertical line though the 0.5 * obar value (same as the horizontal line's value; denote here the vertical line by v and the horizontal line by h), shade the square area inside the rectangle defined by being below v and to the left of h, as well as the area between h and the no-skill line to the right of h. Good probabilistic forecasts will have their scores within these shaded areas. I don't know of any name for h. Finally, I suggested adding the names of the no-resolution, no-skill and perfect reliability lines to the plot, or to the legend for the plot. As it is, it is unclear what each line is; of course, that should be cleared up some with the correct no-skill line added.

TatianaBurek commented 3 years ago

Code spinets from Eric: Let vo = env$o_bar[i][[1]] Assuming the plot region is already set with x- and y- limits from 0 to 1: x.p <- c( vo, vo, 1, 1, 0, 0 ) y.p <- c( 0, 1, 1, (1 - vo) / 2 + vo, vo / 2 ) polygon( x.p, y.p, col = "lightgray" ) ......... abline( v = vo, h = vo, lty = 2, lwd = 1.5 ) abline( 0, 1 ) abline( vo/2, 0.5 )

Also, you can add the writing "No Skill" to the plot using the following: text(0.6, obar + (a-b)(0.6 - obar), "No skill", pos = 1, srt = atan( a - b )/(2pi)*360 )

TatianaBurek commented 3 years ago

Change the names of the lines to Skill->Perfect reliability and Reference ->No-resolution line

dtcenter / METviewer