beyond-all-reason / teiserver

Middleware server for online gaming
https://www.beyondallreason.info/
MIT License
47 stars 47 forks source link

fakedata issue with test user rating deltas - generate them via cumulative sum instead #262

Open jauggy opened 3 months ago

jauggy commented 3 months ago

Here's what it looks like in production (which is correct) 1

my skill went from 22.13 to 23.0 which is a gain of 0.87

But in my dev environment (wrong deltas; ignore the lack of icons which is due to my fontawesome) 2

My skill went from 26.29 to 25.13 which is loss of 1.16 but it shows a gain of 1.18

The deltas are also wrong for both skill, uncertainty, and leaderboard rating. I noticed this bug when I was checking Exeler's last PR to add uncertainty to this page. But then I went back to an older state of master and this bug is still there; so the bug is not from his commit. Something wrong with fake data creation perhaps?

jauggy commented 3 months ago

@Jordan-Cottle can you confirm that you're also getting mathematically incorrect deltas on the match ratings page on your local teiserver?

Jordan-Cottle commented 3 months ago

I noticed that some of the test data for the user did have incorrect deltas. The data in this table comes straight from the database tables, so it seems entirely reasonable that the pre-populated data from the script could just have stuff hard coded that isn't possible to have happen in prod with real data.

Example data from the database: image

Notice it's just json stored in the value column, so really any data could be put in there with no correctness guarantee

StanczakDominik commented 3 months ago

Yeah, the fake data creation iirc basically uses random uncorrelated numbers for both the deltas and the ratings! So they don't make sense. Good find.

One way to improve this would be to use random numbers for the deltas only, and generate the ratings themselves with a cumulative sum over the deltas.