search

Metaculus / metaculus

https://www.metaculus.com
BSD 2-Clause "Simplified" License
52 stars 11 forks source link

Score and medal discrepancies #495

Closed hickman-santini closed 2 months ago

hickman-santini commented 2 months ago

In the Q3 2023 tournament, which should be stable by now, I found differences between the standard platform and the beta. Some of the individual scores, such as mine, differ between the two versions. Also, the original version only gives 42 medals, while the beta shows 43 winners. Relatedly, checking the Real-Time Pandemic Decision Making tournament, it seems that some (such as myself) have gone up in medal ranking (such as from copper to silver). This seems likely to be due to a change in how rounding is handled, but I don't know if it is intentional.

Reported by Sergio

lsabor commented 2 months ago

Great catch to Sergio! It seems like maybe conditional question resolutions weren't successfully migrated.

lsabor commented 2 months ago

Fixed the scoring descrepency - the ArchivedScore migration missed conditional questions so they weren't counted in legacy scored tournaments. Results will take effect after next migration (or if I just call it manually)

The medals earning difference is caused by a known issue where moderators aren't disqualified in the rewrite so they are included in the total count of competitors - so the number of winners (the top 5%) is dependent on this and can effect the results on the margin. This error is strictly good for users, but will eventually be fixed. I'll make that into a separate medium priority task and close this one.

hickman-santini commented 2 months ago

The medals earning difference is caused by a known issue where moderators aren't disqualified in the rewrite so they are included in the total count of competitors - so the number of winners (the top 5%) is dependent on this and can effect the results on the margin. This error is strictly good for users, but will eventually be fixed. I'll make that into a separate medium priority task and close this one.

Confused by this and I don't see the new issue yet so I'll ask here: Do you mean fixing the error will be good for (non-mode) users i.e. move them up in the ranks by disqualifying mods?