search

Sage-Future / fatebook

The fastest way to make and track predictions
https://fatebook.io
MIT License
28 stars 8 forks source link

Brier scores have stayed the same despite many forecasts' resolving #5

Open patbl opened 1 year ago

patbl commented 1 year ago

I created an account and imported my Prediction Book history. I think that since then, the Brier scores displayed by this UI element have stayed the same:

image

My memory is that 0.36 was displayed for "Last three months" even when I initially hadn't made any predictions yet (and all my Prediction Book predictions had resolved more than three months ago).

adam-binks commented 1 year ago

Hi @patbl - I think this was caused by the "last 3 months" looking at when PredictionBook questions were imported, not when they were actually resolved on PredictionBook. I've pushed a fix for this now in this commit.

I think this would cause your "Last 3 months" Brier to stay static if you have lots of "All time" resolved predictions, because a few new predictions won't change it very much. Could you see if it's now fixed?

patbl commented 1 year ago

Would you expect your change to affect predictions that had previously been imported from Prediction Book, or just new imports?

The past-three-months and all-time scores now differ a little bit, but that could be because I've been making and resolving a lot of predictions recently:

image
patbl commented 9 months ago

My date chart still goes back only to August—is this expected to be the case after your fix?

image