search

open-spaced-repetition / fsrs4anki

A modern Anki custom scheduling based on Free Spaced Repetition Scheduler algorithm
https://github.com/open-spaced-repetition/fsrs4anki/wiki
MIT License
2.77k stars 137 forks source link

[BUG] "Ignore reviews before" Not Working #659

Closed anthonysharpy closed 3 months ago

anthonysharpy commented 5 months ago

Describe the bug

When I try to use the "Ignore reviews before" feature it always tells me I have 0 reviews (or a very small number of reviews if I go back far enough), even though I have thousands of reviews.

I have a deck called "Core 2000". Its preset is "Core 2000 Deck". It's a subdeck of a deck called "Japanese". (BTW, this also seems to be not working for all my other decks too, this is just an example.)

image

image

This deck has 100-200 reviews per day:

image

When I don't enter a "ignore reviews before" date, it seems to show the correct number of reviews. For example, this is me pressing "Evaluate":

image

If I set "ignore reviews before" to a recent date - say 1st April 2024 - it will say "Only 0 reviews were found":

image

I have tried changing the filter settings but nothing seems to make a difference.

To Reproduce

Here's an export of my entire collection here, maybe this will help:

japanese.zip

Let me know if I've done the export wrong, never had to do one before...

Expected behavior

There should be thousands of reviews, not 0.

Screenshots

See above.

Environment

Additional context

I've read that this can be caused by not setting up presets properly, but don't think that applies to me (I hope)? https://forums.ankiweb.net/t/failing-to-optimize-evaluate-fsrs-for-some-presets-error-500-only-0-reviews-were-found/43687

I would imagine that if it's able to show me my reviews in the stats graph, it should also be able to figure out my reviews in the optimisation dashboard.

anthonysharpy commented 5 months ago

Interestingly this was opened today https://github.com/ankitects/anki/issues/3242

(edit: personally I've had this issue since I started using FSRS about 3 months ago)

L-M-Sherlock commented 5 months ago

It works as intended. The "ignore reviews before" filters out cards whose first review log's date is before the setting.

anthonysharpy commented 5 months ago

@L-M-Sherlock The description in the app says "If set, reviews before the provide dated will be ignored when optimizing & evaluating FSRS parameters". If that's the intended behaviour then shouldn't it say something like "If set, cards whose first review is before the provided date will be ignored when optimizing & evaluating FSRS parameters"?

L-M-Sherlock commented 5 months ago

The description seems wrong.

brishtibheja commented 5 months ago

IIRC, the reason this feature was added because some people might have changed their grading behaviour at some time in past, in which case those revlogs will be irrelevant, but might not the later reviews still be useful? I don't know why whole cards are being ignored. (the tooltip also mentions the situation where you imported somebody else's decks with their learning history)

@anthonysharpy If you have revlogs from before this particular date you are using for ignore reviews before, how is it possible your first reviews weren't before this date? It shouldn't show "0 reviews found" in my opinion.

L-M-Sherlock commented 5 months ago

I don't know why whole cards are being ignored.

FSRS optimizer doesn't accept incomplete review history.

user1823 commented 5 months ago

I think that the last comment by L-M-Sherlock is slightly confusing. He means to say that FSRS optimizer can't use incomplete review history. The reason is that, in the FSRS algorithm, every review modifies the memory states from the previous one and you can't calculate memory states if you don't have the complete review history.

During reviews, however, we use a hack to estimate the memory states of cards with incomplete revlogs. Otherwise, we won't be able to calculate the intervals. However, this hack is not used when optimizing because optimization is a more high-stakes operation (because optimization affects all the cards not only those with incomplete revlogs).

If that's the intended behaviour then shouldn't it say something like "If set, cards whose first review is before the provided date will be ignored when optimizing & evaluating FSRS parameters"?

It makes sense. You can suggest this on forums.ankiweb.net.

user1823 commented 5 months ago

One correction:

When optimizing, the whole card is ignored. But while evaluating the parameters, only the reviews before the provided date are ignored.

anthonysharpy commented 5 months ago

IIRC, the reason this feature was added because some people might have changed their grading behaviour at some time in past, in which case those revlogs will be irrelevant, but might not the later reviews still be useful? I don't know why whole cards are being ignored. (the tooltip also mentions the situation where you imported somebody else's decks with their learning history)

I am definitely not at all an expert in the way Anki works, but this pretty much sums up what I thought this was for too (and what the description and tutorial seem to say).

If you have revlogs from before this particular date you are using for ignore reviews before, how is it possible your first reviews weren't before this date? It shouldn't show "0 reviews found" in my opinion.

@brishtibheja Might be misunderstanding but my first reviews were before this date (at least for this deck). I guess that would explain why setting it to 1st April causes it to say "0 reviews" - because all my first reviews are from much longer ago.

If it's not possible to optimise using cards with incomplete review histories, I wonder if the ignore reviews before feature actually has any purpose? The tutorial and in-app description seem to say that this can be used for excluding some reviews but not others (which is as the tutorial mentions is useful if e.g. your answering patterns change, or you inherit someone else's deck). But as has been mentioned this seems to not actually be true.

Is it just that this option is useful for evaluating but not for optimising? (Sorry I don't actually know what evaluating is haha)

If this can't be used during optimisation, then maybe this option should be ignored when optimising? And a clearer description added.

If that makes sense then I can go suggest to the folks over on the forums.

user1823 commented 5 months ago

If it's not possible to optimise using cards with incomplete review histories, I wonder if the ignore reviews before feature actually has any purpose?

When a date is set, Anki uses the review history of cards introduced AFTER the date to optimize the parameters. The assumption is that you have at least some cards you started reviewing after the date.

Sorry I don't actually know what evaluating is

Well, this was unexpected because the Evaluate button is placed just to the right of the Optimize button and also explained in the tutorial (which you seem to have read).

anthonysharpy commented 5 months ago

Well, this was unexpected because the Evaluate button is placed just to the right of the Optimize button and also explained in the tutorial (which you seem to have read).

Oh yeah true, brain fart, I do know what it is.

When optimizing, the whole card is ignored. But while evaluating the parameters, only the reviews before the provided date are ignored.

Checking again, I'm not sure this is the case since if I set the date to something like 1st April, it says 0 reviews found when pressing evaluate. So it seems the behaviour is the same (it ignores whole cards) whether evaluating or optimising.

When a date is set, Anki uses the review history of cards introduced AFTER the date to optimize the parameters. The assumption is that you have at least some cards you started reviewing after the date.

That does seem to be the assumption. But I think it's fair to say that that assumption is at odds with what the instructions say at the moment. Sort of seems to me this feature was implemented to solve problem X, but actually does not/can not solve that problem, since the fact that it can only ignore whole cards and not reviews makes the feature a lot less useful. Especially because I think most of the people who will want to use this feature will want to use it on decks that have lots of review history, and decks with lots of review history are less likely to have new cards (unless the user is still adding to it maybe).

user1823 commented 5 months ago

it says 0 reviews found when pressing evaluate

But do you see some values of log loss and RMSE on pressing Evaluate?

If yes, this is another issue (https://github.com/ankitects/anki/issues/3242) where there is a bug in the number of reviews displayed by Anki.

But I think it's fair to say that that assumption is at odds with what the instructions say at the moment.

Suggestions to improve the instructions are welcome. The Anki Forums are a better place for them, though.

since the fact that it can only ignore whole cards and not reviews makes the feature a lot less useful.

I understand what you want to say, but there is no option other than ignoring the whole card.

L-M-Sherlock commented 5 months ago

What's the problem you want to solve via "Ignore reviews before"?

anthonysharpy commented 5 months ago

But do you see some values of log loss and RMSE on pressing Evaluate?

Unfortunately not, it just says "Error: 500: ⁨Only ⁨0⁩ reviews were found.⁩ You must have at least 400 reviews for this operation". (edit: to be clear this is only when I select a recent date, e.g. 1st January 2024)

Suggestions to improve the instructions are welcome. The Anki Forums are a better place for them, though.

Sounds good, I'm about to go on holiday for about a week though, so will probably do it when I come back.

What's the problem you want to solve via "Ignore reviews before"?

Other than the things that have already been mentioned, personally I wanted to see if using only the more up-to-date data (I've had some of these decks for over three years now) might make optimisation more accurate. Not sure how realistic that is, but I was just thinking that the way I answer things might have changed a little bit since then.

user1823 commented 5 months ago

Unfortunately not, it just says "Error: 500: ⁨Only ⁨0⁩ reviews were found.⁩ You must have at least 400 reviews for this operation".

I can't reproduce this. Can you try using the latest Anki version (24.06.2)?

~In https://github.com/ankitects/anki/pull/3106, the minimum limit for Evaluating was removed. Probably, you don't have 400 reviews since 1st January 2024.~

Note that FSRS counts reviews differently. For example, it counts one review when you review the same card on at least two different days.

anthonysharpy commented 4 months ago

@user1823 I get this on 24.06.03 as well

image

Here's a video - am I doing it right?

https://github.com/open-spaced-repetition/fsrs4anki/assets/55365296/4529c0d9-e754-4215-a79b-e05187ab7ab9

Expertium commented 4 months ago

When optimizing, the whole card is ignored. But while evaluating the parameters, only the reviews before the provided date are ignored.

Wait, really? God, this is even more of a mess than I remember. @user1823, can you write a description of "Ignore reviews before" that will be used for https://github.com/ankitects/anki/pull/3288 and in the tutorial? And a description of historical retention. And make sure that both of them don't make the user think "oh no, my reviews will be permanently deleted!". Sorry for the trouble, but I don't know how to write it

brishtibheja commented 4 months ago

Tip (if user1823 would get us a description): Brevity and clarity both are important if we are writing for the tooltips. Thus, write as less as you can but as much as you should.

omer681 commented 4 months ago

I think the issue (judging by the images in the first post) is that the user expects the date convention to be dd/mm/yyyy while it's actually mm/dd/yyyy.

brishtibheja commented 4 months ago

I don't think that's the issue.

anthonysharpy commented 4 months ago

Assuming all the above is true then I imagine the feature should probably also be renamed from "ignore reviews before" to something like "ignore cards learned before"?

user1823 commented 4 months ago

If you forget about Evaluate for a moment, FSRS in Anki has two main components: the optimizer and the scheduler.

As you have correctly interpreted, when a date is set, the optimizer ignores cards learned before the provided date.

However, we can't do this with the scheduler. If the scheduler ignores the whole card, FSRS won't work for that card and SM-2 will instead be used for scheduling. So, as a workaround, the scheduler only ignores the reviews before the provided date and then tries to guess the memory states at the first review done after the provided date.

So, the behaviour of the "ignore reviews before" option is inconsistent between the scheduler and the optimizer. Unfortunately, this means that there is no way to succinctly explain what this option does.

PS: Evaluation behaves exactly the same as the scheduler. This is because the aim of evaluation is to determine how good the scheduler predicts the memory states, which would be most reliable when it works exactly the same as the scheduler.

Expertium commented 4 months ago

Wait, I thought that for scheduling, no reviews are ignored, and cards just use parameters that were generated by the optimizer while ignoring reviews? So the reviews are ignored for the purpose of optimization, but not for scheduling. ...yeah, I really wish someone other than me wrote that section of the tutorial, I have absolutely no clue how any of this works

Expertium commented 4 months ago

Ok, so there are only three people in the world who understand how "Ignore reviews before" works: @L-M-Sherlock, @user1823, and @Luc-Mcgrady. Unfortunately, LMSherlock's English isn't good enough to write the tutorial and the tooltip for Anki. That leaves user1823 and Luc. Can either of you guys help with this, this and this, please?

Luc-Mcgrady commented 4 months ago

As has been explained in this thread and for reasons that were explained to me at length which I have since somewhat forgotten, this option ignores cards which are introduced after the date set. (I implemented it the way it is described at first but was told it would mess with the first review which is very important for FSRS).

This means if a card does not have a learning step after the given date, all reviews for that card are completely ignored.

This, as I see it, leaves people with 2 options:

The manual should really read like this

If set, 
- reviews before 
+ cards which have no learning steps after
the provided date will be ignored when optimising & evaluating FSRS parameters. This can be useful if you imported someone else's scheduling data, or have changed the way you use the answer buttons.

I suppose the option should be named "Ignore cards before"

Expertium commented 4 months ago

I suppose the option should be named "Ignore cards before"

I'm afraid that will make users think that cards that were created before that date will be ignored.

Luc-Mcgrady commented 4 months ago

"Ignore cards introduced before"?

Expertium commented 4 months ago

"Ignore cards reviewed before" then, though it's a bit too long

Luc-Mcgrady commented 4 months ago

"Ignore introduced before"?? little less clear

brishtibheja commented 4 months ago

I think this discussion belongs here: https://forums.ankiweb.net/t/ignore-reviews-before-is-misleading/46686

Dae and others in the forum can also chip in.

L-M-Sherlock commented 3 months ago

This issue has been inactive for a while. Let me summarize it:

  1. The previous description of "Ignore reviews before" is misleading. Now it has been fixed in https://github.com/ankitects/anki/pull/3314/commits/495b07a05ee8c4548ecbe3ff4be900930de915e1
  2. The current implementation cannot only ignore reviews instead of cards unless I refactor the training process of FSRS in the backend to support sample weight.

I have no plan to support it, so I close this issue.