search

yhoiseth / python-prediction-scorer

Python library to score predictions
https://pypi.org/project/predictionscorer/
MIT License
6 stars 0 forks source link

Distance scoring rule #77

Open yhoiseth opened 4 years ago

yhoiseth commented 4 years ago
Screen Shot 2020-08-16 at 17 01 32
yhoiseth commented 4 years ago

Two things about the distance scoring rule are unclear to me:

  1. The probability is a constant (β). Why is that? Does it mean that you can’t use the rule to compare confidence intervals that have different confidence (probability)?
  2. The probability is not taken into account when outcome (x) is inside the interval. Why is that? Intuitively, I would think that the probability should matter in those cases, too.
yhoiseth commented 3 years ago

Asked Spencer. Answer:

Regarding (1), if I recall correctly, when I wrote this I recall being unconvinced that the standard approach to doing interval predictions (the linear method) actually supported comparisons across different betas properly. I recall trying to improve that for the distance scoring method, but I cannot promise it works in an ideal way. But my memories on this are somewhat hazy.

Regarding (2), when beta gets bigger you want the forecaster to be incentivized to use larger intervals (a 99th percentile interval should be larger than a 50th percentile one). If I recall correctly, the way I incentivize this is to have the penalties for being wrong (having the true value fall outside your interval) go up when beta increases. Another way to achieve a similar goal would probably to have the reward for being “right” go up as beta increases with the penalties held constant. It may be I went with the first method due to wanting to have a consistent max score independent of beta, but I’m not sure.

I also asked him about c:

There is this variable c that “determines the units with which we measure distance”. It is not obvious to me what this means. For now, I have just set c = 1. Do you have an example that could make this easier to understand?

Answer:

Think about it like this: if we’re predicting distance (say, the length of a tennis court) are we predicting in feet or meters or miles? c just accomplishes a unit conversion in essence, I think. So if the user inputs in feet you can have c convert to meters before being scored.

More generally, it encompasses the idea of what you need for something to be a “good guess”. If in miles being off by 0.5 miles is probably a good guess. If in feet then being off by 0.5 miles is probably a terrible guess.

yhoiseth commented 3 years ago

I think I’ll leave out c as it is not obvious to me that it is useful in this case.