search

Metaculus / metaculus

BSD 2-Clause "Simplified" License
37 stars 5 forks source link

Beta calibration curve looks different from www #480

Open SylvainChevalier opened 3 weeks ago

SylvainChevalier commented 3 weeks ago

beta: image

www: image

Let's check the maths, @lsabor.

lsabor commented 3 weeks ago

Somewhat fixed: the displayed Beta one is for the metaculus prediction, here is the recency-weighted community prediction version: image

here is the updated rewrite version: image

lsabor commented 3 weeks ago

I've moved this to medium priority. @SylvainChevalier feel free to move it back to high priority.

SylvainChevalier commented 3 weeks ago

As discussed yesterday, the top bin being bad and different is concerning.

George3d6 commented 3 weeks ago

@SylvainChevalier are the bucket the same here ? it looks like the top is different in the new one (100 vs 95)

lsabor commented 2 weeks ago

Update: Recency Weighted main site: image

Recency Weighted rewrite: image

MP main site: image

MP rewrite: image

lsabor commented 2 weeks ago

Here's what happens when I set the forecast horizon to be from open to actual close time not scheduled close time: image

What's going on here is that "coverage" in the main site for ComboPrediction is 100% if a forecast starts at open even if the question resolves early. Looks like that was our culprit...

lsabor commented 2 weeks ago

Well, that's definitely not the whole difference as my personal track record is still off: image

image

Note the 4th and 6th bins are much closer in the main site than the rewrite.

lsabor commented 2 weeks ago

merged above changes with notes. still more to do though