probabilistic forecast metrics: add quantile score

[dplarson]

This issue is to discuss adding the quantile score to the list of probabilistic forecast metrics.

Given a set of observations (obs_1, obs_2, ..., obs_n), forecasts (fx_1, fx_2, ..., fx_n}), and corresponding forecast probabilities (p_1, p_2, ..., p_n), the quantile score is defined as:

• 1/n * sum_{i=1}^n (obs_i - fx_i) * (1{obs_i >= fx_i} - p_i)

where 1{obs_i >= fx_i} an indicator function (1 if obs_i >= fx_i, 0 otherwise).

The quantile score is similar to the Brier Score (1/n * sum_{i=1}^n (p_i - 1{obs_i >= fx_i})^2), but where the Brier Score is unitless (values between [0, 1], the quantile score returns the same units as the observations (e.g. power in MW, then quantile score in MW).

Here's an arbitrary example to illustrate:

• forecast power generation at time t as 4 MW with 50% probability: fx = 4 MW, p = 0.5
• actual power generation at time t is 5 MW: obs = 5 MW
• Brier score = (0.5 - 1{5 MW >= 4 MW})^2 = (0.5 - 1)^2 = 0.5^2 = 0.25
• quantile score: (5 MW - 4 MW) * (1{5 MW >= 4 MW} - 0.5) = (1 MW) * (1 - 0.5) =0.5 MW

Arguments in favor of adding quantile score:

• the metric is commonly used in quantile regression (dating back to a 1978 paper by Koenker and Bassett)
• the metric is readily computed and compatible with solarforecastarbiter.metrics.probabilistic and related code (i.e. there would be minimal code change required to add support)

Arguments against adding the quantile score:

• the metric provides similar information as the Brier score

Attached is the paper from Koenker and Bassett (1978) that introduced quantile regression and a more recent paper from Bouallegue, Pinson and Friederichs (2015) that adds additional discussion of the quantile score.

Koenker and Bassett (1978): koenker-bassett.pdf

Bouallegue, Pinson and Friederichs (2015): qj.2624.pdf

[wholmgren]

Thanks for the references. Pros outweigh the con. Also has a nice connect to the cost/loss framework. Do we also need a Quantile Skill Score?

[dplarson]

Adding Quantile Skill Score also makes sense.

As a next step, I'll open a PR to add Quantile Score and Quantile Skill Score. I expect the PR will be primarily additions to solarforecastarbiter.metrics.probabilistic, but I'll also check that the report code works with the new metrics.