add more references to metrics

[wholmgren]

following example introduced in #108. Start with the references section of the April survey doc.

[dplarson]

What's the preferred convention for citing the references in the metrics page? The example in #108 used just the last name of the first author, but it might be better to use author-year or numbers to avoid confusion for multiple references with the same first author (e.g. "Espinar (2009)" or "[1]").

[cwhanse]

I'd vote for AuthorYY(a, b, c,) e.g. Espinar08, Espinar08a, Espinar08b, but I don't have a strong preference here.

[wholmgren]

Either format using the author name and year is ok with me. Numbers only might be more difficult to keep track of in this kind of document.

[dplarson]

Author-year sounds good to me. I'll update PR #110 accordingly.

[dplarson]

Of the metrics listed on the website, the obvious ones that need references are:

• [x] forecast skill: Marquez and Coimbra (2012) "Proposed Metric for Evaluation of Solar Forecasting Models"
• [x] OVER: Espinar et al. (2009)
• [x] CPI: Gueymard (2012)
• [x] Brier score: Brier (1950) "Verification of forecasts expressed in terms of probability"
• [x] REL, RES, UNC (decomposition of Brier score): Murphy (1973) "A New Vector Partition of the Probability Score"
• [x] CRPS: Matheson and Winkler (1976) "Scoring Rules for Continuous Probability Distributions"

EDIT: I'm editing this comment with the "original" references for each metric.

[wholmgren]

Given the discussion in https://github.com/SolarArbiter/solarforecastarbiter-core/issues/233 I suggest including a second reference for the Brier score and adding some discussion about how we only consider the more commonly used binary version of the score. Obviously I'm partial to Wilks for a second reference.

CPI is Gueymard CA. Clear-sky irradiance predictions for solar resource mapping and large-scale applications: improved validation methodology and detailed performance analysis of 18 broadband radiative models. Sol Energy 2012;86:2145–69.

[dplarson]

With PR #117 merged, this issue can be closed.

[dplarson]

Since this issue is still open:

We should add references to the deterministic event forecast metrics section, e.g.,

• Jensen, T. L., Fowler, T. L., Brown, B. G., Lazo, J. K., & Haupt, S. E. (2016) "Metrics for Evaluation of Solar Energy Forecasts", (No. NCAR/TN-527+STR). doi: 10.5065/D6RX99GG
• Chapter 8 "Forecast Verification" of D. S. Wilks "Statistical Methods in the Atmospheric Sciences"

[wholmgren]

for the EBIAS metric, the Wilks discussion was helpful to me:

The bias is simply the ratio of the number of “yes” forecasts to the number of “yes” observations. Unbiased forecasts exhibit B = 1, indicating that the event was forecast the same number of times that it was observed. Note that bias provides no information about the correspondence between the individ- ual forecasts and observations of the event on particular occasions, so that Equation 8.10 is not an accuracy measure. Bias greater than one indicates that the event was forecast more often than observed, which is called overforecasting. Conversely, bias less than one indicates that the event was forecast less often than observed, or was underforecast.

we don't want to overdo it on our page, but maybe you can add a little more than what we currently have?

[dplarson]

we don't want to overdo it on our page, but maybe you can add a little more than what we currently have?

I like this suggestion. I'll open a PR for adding the references and a more descriptive (but still concise) explanation of the EBIAS metric.