wasabipesto / themis

A prediction market calibration suite
https://calibration.city
MIT License
9 stars 1 forks source link

Possibly using incorrect starting forecast date for Metaculus forecasts #12

Open ryooan opened 1 day ago

ryooan commented 1 day ago

Looking at the way it's grabbing Metaculus questions, it looks like it's using "created_time", which is the time when an author submitted the question. This is the time the question draft was first saved, and in many cases a question wouldn't actually open for forecasting until later on. I believe it should instead use the date the question became open for forecasting, the publish time. This may be skewing the probabilities pulled for Metaculus questions, because it looks like created_time is being used for open_dt which is used to get the probabilities at different points. Especially historically, many questions on Metaculus are created by a user as drafts, later on submitted to the pending queue for review by moderators, and then reviewed and set to upcoming before they are eventually open to forecasting. This process in the past has taken as long as months for some questions, so it could potentially cause large differences in the forecasts if I'm understanding correctly.

image

image

image

(Note in the new api I believe the corresponding times to those mentioned above are "created_at" and "open_time")

wasabipesto commented 1 day ago

Good catch, I will check this and fix.