search

open-spaced-repetition / srs-benchmark

A benchmark for spaced repetition schedulers/algorithms
https://github.com/open-spaced-repetition/fsrs4anki/wiki
62 stars 9 forks source link

A new way to visualize performance.py #113

Closed Expertium closed 1 month ago

Expertium commented 1 month ago

I added a new table to visualize performance of each algorithm. It shows the % of users, for whom algorithm A (row) has lower RMSE than algorithm B (column). For example, 95% means that for mor users, algorithm A results in lower RMSE, and only for a small minority of users, algorithm B results in lower RMSE. I'll also edit the README.

Expertium commented 1 month ago

Okay, well, I don't know how to modify multiple files in the same PR https://github.com/open-spaced-repetition/srs-benchmark/pull/114

Expertium commented 1 month ago

I just realized that there is a problem. If in algo1.jsonl the data is like this:

user1: 0.5
user2: 0.4
user3: 0.3
user2: 0.2

But in algo2.jsonl the data is like this:

user1: 0.5
user3: 0.3
user3: 0.4
user2: 0.2

The pairwise comparison will be inaccurate. In other words, everything has to be sorted by user. I'm currently trying to fix it, I'll either make another PR or ask for your help later.