Converting from Anki SM-2 to FSRS: Rescheduling gives a lot of cards?

[kuroahna]

Anki Version 2.1.55 Beta 2 FSRS Scheduler: v3.4.0 FSRS Optimizer: v3.5.0 FSRS Helper Addon: v3.4.0

I'm rescheduling all my cards to use the FSRS algorithm now, but I'm not 100% sure if this is intended or if there's perhaps a bug?

1. Open Anki
2. Click File -> Switch Profile
3. Click Add
4. Name it Testing (or any other name)
5. Click ok
6. Import my collection collection2.zip
7. See 499 cards due
8. Click Gear icon for the 日本語 deck
9. Click Options
10. Add Custom scheduling code with my parameters (v3.4.0) scheduler.zip Note I've only changed the following parameters

    var w = [1.3027, 0.5276, 5.1668, -1.3924, -1.0168, 0.0089, 1.2972, -0.0173, 0.6662, 1.8265, -0.4096, 0.688, 0.4756];
    let requestRetention = 0.86;

11. Click Save
12. Click Tools -> Reschedule All Cards
13. See 7615 cards due

Perhaps this is intended and I really do need to review all 7615 cards due to my parameters, but I'd like some confirmation whether or not this is expected behaviour?

I've also noticed some interesting values with the customData after rescheduling.

1. Right click anywhere in anki in the deck screen
2. Click Inspect (Install webview inspector addon)
3. Click the 復習 filtered deck
4. Click Rebuild
5. Click Study Now
6. Anki shows me the card with the Word 女中 (card id is 1648356923674)
7. Click Browse
8. Right click highlighted card (it is 女中 for me)
9. Click Info...
10. See these values Note that lapses is 0 (I resetted the card before on 2022-07-05 by right clicking the card and clicking Forget Card..., which resets the ease factor + lapses count)
11. In the WebView Inspector, I see the values

Note that interval=60. This means that 60 days have been elapsed since my last review, which was on 2022-08-17 for this card, and Anki SM-2 schedules it in 2.37 months from 2022-08-17, which should be about in 72.08 days, or on 2022-10-28. However, rescheduling the card using FSRS Helper assigns it to me today instead. I assume this is because of my parameters? However, it is interesting to see that last_s=11.806 and last_d=9.8721 and retrievability=0.585400431176371

Shouldn't last_s be a much higher value than 11.806? From my review history, when I resetted the card on 2022-07-05 and relearned the card, I've always pressed Good on the card and haven't failed the card. Of course, before 2022-07-05, I've failed the card 4 times before resetting the card. In Anki, I set Lapse Threshold set to 4 and Auto Suspend the card. This allows me to find leeches fast, which forces me to reformulate the card and make it easier for me to understand. Then I forget the card and relearn it as if it was completely new. Does this affect how the FSRS optimizer/scheduler/helper addon works? Is this why the last_s is 11.806 instead of a much higher value? Similarly for last_d? Shouldn't last_d be a much lower value than 9.8721 since I haven't failed the card at all yet? I feel like last_s and retrievability should be much higher values for this card, and last_d should be a much lower value.

Perhaps this is also why the helper addon is scheduling me +7000 extra cards? It'd be much more understandable if it was maybe ~1000 cards or so, but ~7000 seems quite a lot

In the code, https://github.com/open-spaced-repetition/fsrs4anki-helper/blob/6e58aa90e786f7581e9db12346cb5a6052792669/reschedule.py#L87-L111 I see that it loops through the revlog history and replays the memory states, which takes into account the revlog history before I resetted the card. Maybe it should start from the latest reset point? It'll probably need to loop through the revlog in reverse (starting from the end), looking for when it was last reset, then it'll start going through the memory states with that as the starting point. Otherwise, if the card wasn't reset at all, then we use the whole revlog as previously done.

Also, another question, for the Optimizer, does it make sense to handle this case as well? Should we ignore all the data points in the revlog before the card reset since it's no longer "valid" data? Or does it make sense to keep it because if we ignore it, we'll have bias in the data set? I'm not too knowledgeable in this field, but just wanted to get your thoughts on this.

[L-M-Sherlock]

What does the manual reset mean? Do you modify the content of the card? I think using the entire history is reasonable if you didn't change the card's content.

[kuroahna]

What does the manual reset mean?

Manual reset happens when you

1. Right click the card
2. Click Forget...
3. Check Reset reptition and lapse counts
4. Click Ok
5. Right click card again and click Info...
6. 

Do you modify the content of the card? I think using the entire history is reasonable if you didn't change the card's content.

Yes, I modify the contents of the card. Previously, the card was just a vocabulary card, meaning that there's only the word on the front, like this:

However, since the card has leeched (auto suspend the card after failing the card 4 times), I modify the card to make it easier for me to review the card. I do this in a variety of ways, but a common way is that I add a hint to the front of my card, where I add an example sentence and possibly the reading (onyomi or kunyomi) to remind me of the individual kanji readings like this

In other words, I changed my vocabulary card into a sentence card. In my opinion, this means that the card is no longer the same difficulty as it previously was (showing only the word on the front)

Is this a valid use case?

If not, I could just not use the helper addon and only use the scheduler. It will convert the ease factor + interval into difficulty+stability for me.

[kuroahna]

Might be also worth noting that if you

1. Right click the card
2. Click Set due date
3. Put 7 in the input box
4. Right click the card and click Info

5. 

You will see it will also set the Type to Manual.

I personally don't use this feature, but I know there might be some users out there who manually change the due date of their cards.

If you do end up supporting this use case, we'll need a way to differentiate when a user resets a card (they click the Forget button and select the reset repetition and lapse counts option) with when a user manually changes the due date of a card (they click the Set Due Date button), or else we might have bugs in the code.

[L-M-Sherlock]

I will study the difference between reset repetition and lapse counts and Set due date.

[L-M-Sherlock]

I fix a bug related to this issue at the newest version.

It will not reschedule a card twice.

https://github.com/open-spaced-repetition/fsrs4anki-helper/blob/d31b7e949a16e2ecde096656b044b1c833ab8335/reschedule.py#L93-L96

And distinguish the set due date and forget card.

https://github.com/open-spaced-repetition/fsrs4anki-helper/blob/d31b7e949a16e2ecde096656b044b1c833ab8335/reschedule.py#L104-L113

[kuroahna]

Thanks, I just tried it and it works. I get ~6500 cards instead of ~7500 now, but I think that's because it's rescheduling a lot of my very old cards which are bad.

One example is my card with the word 絡める (cid is 1588050731577)

1. Import my collection
2. Import my FSRS scheduler settings shown in the original post above
3. Reschedule all cards
4. Tools -> Create filtered deck
5. Set Word:絡める as filter
6. Click rebuild
7. Click the filtered deck to view the card

I get

interval=502
lapse=2
last_s=113.3371
last_d=9.6466
retrievability=0.6270872472636866

I feel like these values are still wrong somehow. last_s is way too small. This card is very easy for me now, retrievability should probably be much higher than 62.7%.

I'm using the FSRS Optimizer to simulate my review history for this card

test_rating_sequence = "1,3,3,3,1,2,2,3,3,3,3,3,3"
requestRetention = 0.86  # recommended setting: 0.8 ~ 0.9
easyBonus = 1.3
hardInterval = 1.2

...

(tensor(1.3027), tensor(7.9516))
(tensor(2.8282), tensor(7.9268))
(tensor(5.8703), tensor(7.9023))
(tensor(11.9317), tensor(7.8779))
(tensor(4.2065), tensor(9.8693))
(tensor(5.6691), tensor(10.))
(tensor(7.3734), tensor(10.))
(tensor(9.4018), tensor(9.9570))
(tensor(12.7977), tensor(9.9144))
(tensor(17.6543), tensor(9.8721))
(tensor(24.6099), tensor(9.8302))
(tensor(34.6341), tensor(9.7887))
(tensor(49.3470), tensor(9.7476))
rating history: 1,3,3,3,1,2,2,3,3,3,3,3,3
interval history: 0,2,4,8,17,6,7,8,13,18,25,35,50,71
difficulty history: 0,11.4,11.3,11.3,11.3,14.1,14.3,14.3,14.3,14.2,14.1,14.1,14.0,14.0

But I'm not getting the same difficulty, and the interval history doesn't quite match the last_s value (says 71, but last_s says 113)

If I reimport my collection, scheduler settings, and instead let the FSRS scheduler convert the ease_factor/interval to difficulty/stability, then I get the following values

interval=502
lapses=2
last_d=4.36
last_s=564
retrievability=0.9104845657918987

This values seem to make a lot more sense to me.

Another example I have is my card 改装 (cid is 1465371045396)

There's a lot of issues with this card... For example, on 2019-06-12, I learned the card and the starting ease is 180%. I press Again which drops ease to 160% and then eventually 130% since I keep pressing Hard. But eventually I learn about ease hell and then use an addon to reset all my card's ease to 250%. So this card's ease went from 130% to 250% magically.

Another issue is this card somehow jumps from 6 day interval to 1.98 years interval. I do not know how.

Either way, I have quite a lot of cards that are bad like this, which is probably why FSRS helper addon gets messed up for these kinds of cards. I think I should not use the helper addon, since a lot of my old cards are like this. I don't think the addon can fix this, which I'm okay with. This is my fault

I think we can close this issue unless there's something else that I missed?

[L-M-Sherlock]

Thanks for your feedback. I find more bugs in your case (T_T). Your case is worth further research.

[L-M-Sherlock]

Could you share the collection with me? You can upload the file here: https://forms.gle/KaojsBbhMCytaA7h8

[kuroahna]

I shared my collection in the original post.

I can reupload again here: collection2.zip

Or do I need to upload it through the Google Forms?

[L-M-Sherlock]

It is OK. I missed it.

[L-M-Sherlock]

Another example I have is my card 改装 (cid is 1465371045396)

In this case, you forgot too much in 2019-11-01, 2019-11-19, 2019-12-02, and 2019-12-11. I think the interval given by FSRS is more reasonable. The factor of FSRS would not increase without limit, but the interval given by Anki's built-in schedule linearly depends on the delay.

[L-M-Sherlock]

One example is my card with the word 絡める (cid is 1588050731577)

I found that the difficulty increases too fast. Maybe I need to improve the formula for updating difficulty.

[kuroahna]

In this case, you forgot too much in 2019-11-01, 2019-11-19, 2019-12-02, and 2019-12-11. I think the interval given by FSRS is more reasonable. The factor of FSRS would not increase without limit, but the interval given by Anki's built-in schedule linearly depends on the delay.

I think I remember what happened with this card. This was when I started learning Japanese years ago. One of the first few cards I was learning and it was difficult for me at the time. Last review was 2019-12-11. Then I stopped reviewing the deck. And then I came back on 2021-06-12 and was able to remember the word, which is why it gave me a 1.98 interval. I forgot that this is a feature of Anki.

I found that the difficulty increases too fast. Maybe I need to improve the formula for updating difficulty.

That'll be interesting to see! Running the optimizer v3.7.1, I get the following difficulty distribution

difficulty
1     0.006349
2     0.004866
3     0.005231
4     0.008408
5     0.133489
6     0.033219
7     0.042266
8     0.221167
9     0.028195
10    0.516812

It's interesting to see that a large majority of my cards are at difficulty level of 10, and barely any below 5. I think this is due to the way I review a majority of my cards in Anki SM-2.

I have learning steps 1 5 60 in Anki. And when I learn new cards in Anki, I always press Again so that it goes through all 3 steps, even if the card is "easy". I've recently changed my learning habits, so that if the card I'm learning is something I "already know", then I just press Good, so that it will skip the first learning step and go to 5 60 immediately.

And I also never press Hard or Easy to avoid "ease hell" (binary grading system). So my difficulty never drops below 5 using FSRS. I think the cards that are in the 1-5 difficulty group are my really old cards when I did use to press Easy.

Also, I set my leech threshold to 4 and auto suspend my cards. Then I go through my leeches and change the vocabulary card into a sentence card, reset the card, and relearn it. So the card is now "new" and the old revlog doesn't apply for that card anymore. I think the optimizer is still using the old revlog values

So this is probably why my difficulty distribution is heavily skewed towards 5-10, with about 51.6% being a difficultly of 10, since it's probably not ignoring the old revlog for the cards I've reset, and that I always just press Good and Again only

[L-M-Sherlock]

Also, I set my leech threshold to 4 and auto suspend my cards. Then I go through my leeches and change the vocabulary card into a sentence card, reset the card, and relearn it. So the card is now "new" and the old revlog doesn't apply for that card anymore. I think the optimizer is still using the old revlog values

Thanks for finding the bug. I will fix it.

[L-M-Sherlock]

I found it tricky to deal with the manually rescheduled cards in the review logs. There are only 2000+ logs affected in your collection. I assume the burden was not induced by it. So I try to analyze the collection. The result shows that your retention is lower than 80% when the first rating is again. I guess it caused the helper to give a lot of cards after rescheduling.

1:again, 2:hard, 3:good, 4:easy

      r_history  avg_interval  avg_retention  stability  factor  group_cnt
              1           1.0          0.922        1.3     inf      17058
            1,3           3.8          0.824        2.1    1.62      15545
          1,3,3          11.2          0.800        5.4    2.57      12477
        1,3,3,3          31.6          0.805       16.0    2.96       9090
      1,3,3,3,3          80.9          0.847       56.5    3.53       5177
    1,3,3,3,3,3         104.6          0.764       45.0    0.80       1764
  1,3,3,3,3,3,3         214.2          0.775       92.5    2.06        409
              3           1.0          0.926        1.4     inf       8138
            3,3           2.9          0.948        6.4    4.57       6429
          3,3,3           6.1          0.977       25.7    4.02       4821
        3,3,3,3          12.9          0.964       35.0    1.36       3617
      3,3,3,3,3          29.3          0.940       46.3    1.32       2366
    3,3,3,3,3,3          51.0          0.925       65.0    1.40       1229
  3,3,3,3,3,3,3          36.8          0.964       97.3    1.50        582
3,3,3,3,3,3,3,3          86.1          0.826       48.1    0.49        107
Analysis saved!

[L-M-Sherlock]

Now you can alleviate this problem via the postpone feature.