search

Metaculus / metaculus

https://www.metaculus.com
BSD 2-Clause "Simplified" License
54 stars 12 forks source link

Implement range questions with discrete values #1233

Open malginin opened 2 weeks ago

malginin commented 2 weeks ago

For many range type questions, the set of possible outcome values is discrete. Current examples: https://www.metaculus.com/questions/28845/how-many-of-the-q3-2024-top-16-will-be-in-top-16-in-q4-2024/ https://www.metaculus.com/questions/29028/us-state-dept-global-arms-sales-approved-q4-2024/ (Here, the outcomes are integers but often the resolution source contains numbers rounded to 1 or 2 decimal points, which also means the set of outcomes is discrete).

With the new ability to add an unlimited number of components, to increase their scores, the more experienced forecasters now create predictions with dozens of narrow components centered on the possible discrete values. This requires an expensive time investment and disadvantages the forecasters who are less familiar with the Metaculus interface and scoring system.

Desired behavior: For the questions with discrete outcomes, use a probability mass function instead of a probability density function and have each component represent a binomial distribution instead of a normal one.

lsabor commented 2 weeks ago

Interesting idea. I think an alternative here would be to add a setting on each Question that defines the number of bins to consider. For questions with only discrete outcomes (or let's just say evenly spaced expected outcomes that are less than 30 or so in number), the question writer would determine exactly how many "buckets" to have within range.

To let you in on 2 secrets (not actually secrets, just little known facts):

  1. the "pdf" is actually a "pmf" with 200 values (plus 2 additional values for below and above bounds). For questions with a medium number of possible outcomes (too many for a Multiple Choice question), we could just lower that number.
  2. the API actually allows you to predict directly with a 201 point cdf, so you can push the boundaries even further when it comes to making an extremely precise forecast. This is definitely an advantage for those savvy enough to use the API, and so we intend to make it a little easier to understand and eventually add more ways to forecast on continuous questions. We actually have just started supporting forecasting with percentiles to support continuous questions for the AI benchmarking tournament, but it's not yet documented.
lsabor commented 2 weeks ago

Assigning Sylvain to this just for visibility so we can discuss the possibility.

SylvainChevalier commented 2 weeks ago

We've considered this new question type in the past. It is only necessary for questions with 10-30 options, since below that you can use a multiple choice, and above that you can use a continuous.

A recent development is our lifting the maximum number of components you can use on continuous questions, making this abomination possible.

On balance, this new question type would be nice to have, but would only somewhat improve a small number of questions. It's unlikely we'll prioritise working on it above our other endeavours in the near future.

Did you know we very much welcome contributions, now that we're open source? We'll even send you a very nice Metaculus hoodie.

ryooan commented 1 day ago

If this isn't too difficult to do I'd like to put a vote in for someone working on it in the near-ish future. It would be really valuable for group questions like these: https://www.metaculus.com/questions/13960/us-monthly-headline-cpi-inflation/

The ideal for that one would be a group of multiple choice questions, though we currently are unable to do that. But would be much nicer to be able to just fit a distribution and specify that it's going to resolve to a precision of 0.1, and any mass between 0.1 increments should be rounded to the nearest increment. Currently these are tedious to predict on and it's probably hampering engagement (any question where you have to make more than 1 or 2 spikes to have an optimal forecast is very tedious in general).

malginin commented 1 day ago

Currently these are tedious to predict on and it's probably hampering engagement (any question where you have to make more than 1 or 2 spikes to have an optimal forecast is very tedious in general).

I can confirm as a forecaster: I opened that question and immediately ran away screaming.