Review cards by reading text and some other small changes

[dayjaby]

I made quite a lot of changes, without separating them well in different commits. I hope it doesn't become too troublesome.

First of all, I reverted some code of the last pull request. Instead of having a {translation} tag, we have now {line}, {t0},{t1},{t2} and so on. {t0} is the first tab-separated field in the line, {t1} the second one and so on. Apart from that I added a tag {filename}, which contains the path of the opened file.

Instead of preventing users to not add a note for a vocabulary that exists already, he has the option to overwrite it now. Overwritting is done by clicking on the blue '+'-button. In this case, all fields in the existing notes are overwritten. Except for the case that the name of a model's field ends with '+'. In this case field's new value is added to the old value with '
' in-between.

Usage example: My model has a field 'Examples+' with the value '{line} (<a href="file:///{filename}">File</a>)'. The Examples+ for the note 幸い (in Matthew 5) would look like this: 3 「心の貧しい人たちは、幸いである、天国は彼らのものである。 (File) 4 悲しんでいる人たちは、幸いである、彼らは慰められるであろう。 (File) 5 柔和な人たちは、幸いである、彼らは地を受けつぐであろう。 (File)

Added a support for saving files.

So far Yomichan handles 々 occurences not at all. Examples: 人々,様々. This is now handled during the sanitizing. It merely copies the previous kanji.

Added a preference option to immediately 'unlock' vocabulary. The meaning is that when I add a new vocabulary to my deck, it shall be put immediately out of the new queue into the learning queue.

These were the minor changes. The major change is that it's now possible to review Anki cards merely by reading the text with Yomichan. While reading, just lookup the definitions of words as you are used to do. This way, the vocabulary is added to the 'wrong' list. After completing a text, you click on the button 'Learn vocabulary'. It gives you a notification like "189 correct and 17 wrong". Every text will have a vocabulary list. Whenever you add or overwrite a note, it gets added to the vocabulary list. On saving a text, the vocabulary list and the text are separated by a line "### VOCABULARY IN THIS TEXT ###". I uploaded an example here.

Because we read a lot of undue cards, whereas Anki doesn't allow early reviews of them (or rather: Anki would calculate the new times as if the card is due), I made a custom scheduler, which handles these cases properly. For example if I read today one text with the word 私 and another text with 私 the following happens: After reviewing text 1 私 is rescheduled. After text 2 私's times aren't changed, because interval time = (due date - today). If (due date - today) is half of the interval time (the vocabulary is half-due), then the change is 50% of what it used to be in Anki.

Ok, that's all!

I consider this to be quite a revoluationary approach to vocabulary learning. A further enhancement would be to have a list of text files with the amount of due cards in each (where half-due cards are counted as 0.5). That way a student can focus on re-reading texts with a lot of due cards.

[FooSoft]

Looks pretty sweet on a first glance. I will be looking over this over next couple of days to see if I can find any issues or anything to knitpick about : )

[FooSoft]

One small thing: I have a updates file that is used to automatically notify users about changes to Yomichan when they run the plugin: http://foosoft.net/projects/yomichan/dl/updates.json

If you can provide a concise feature list, that would be swell.

[dayjaby]

The biggest problem with this update is that a lot of things happen under the surface. A new user might accidently click on "Learn vocabulary" and non-expected things occur. Right now I use the "Anki" panel to display the vocabs of a text. But it's not clear that all these are actually learned after clicking on "Learn vocabulary". Furthermore there is no extra list of "bad" vocabulary. And the user should have the option to remove vocabulary out of the list.

Or on saving text files he might be surprised that there is some kind of vocabulary list attached. So maybe we could have a "Save..." menu item and a "Save file and vocab..." menu item to make it clear.

And overwriting hiragana-only vocabulary is not implemented yet (the one where you click on the small "+"). Edit: DONE

And if you lookup a vocabulary, not all vocabulary should be added to the bad list, but maybe only the longest match. So if you lookup 踏み抜く, then 踏む shouldn't be added to the bad list. And there should probably be a status message at the bottom like "踏み抜く has been put into the wrongly answered vocabulary set" Edit: DONE

Another critical point: The statistics of Anki are distorted heavily by the usage of this. At the moment each card is answered in 0 seconds. Something like "Average answer time: 0.4s (153.1 cards/minute)" contains no information at all. If we find a good way to time the vocabulary, then it would be possible to compare this method to the usual one. Like "Oh, now I only need 2 seconds per card instead of 10. That's awesome" ;)

Ok, so far my self-criticism.

[dayjaby]

With the newest commit, it's possible to store the text files in collection.media\Yomichan and these text files get automatically added to the deck list in Anki. That way it's possible to quickly decide which text contains a lot of due vocabulary. A benefit of using collection.media is that the text files can be synchronized as well! Clicking on such a text deck causes Yomichan to open the corresponding file.

Some implementation notes:

• Adding decks to the tree can be done with a custom Scheduler. But each text deck still needs an unique existing deck id. Thus new decks are created, but they remain empty.
• Whenever the deck browser opens, all text files are parsed and the dueness is calculated. As before mentioned, cards don't have to be 100% due. A 50% card is considered as 0.5 due cards.

[FooSoft]

Coolness, I'm going over the changes locally and making a couple of small tweaks here and there.

[dayjaby]

Good, but make sure to use the latest commits. I tried to put everything in its proper place. For example the custom scheduler in anki_bridge.py instead of reader.py.

[FooSoft]

Yeah, I'm pulling from your master branch. A problem I found is that there is a bunch of Anki stuff inside of yomichan.py. One goal of the way Yomichan is set up is to keep all of the Anki related things abstracted inside of anki_bridge.py -- this way if and when version updates break Yomichan, it's possible to isolate the changes to just one module. Furthermore, everything non-Anki related should work fine just by running yomichan.py (basically running outside of Anki environment). Right now when I try this I get

alex@flatline /m/s/p/yomichan> ./yomichan.py 
Traceback (most recent call last):
  File "./yomichan.py", line 26, in <module>
    from aqt import mw
ImportError: No module named aqt

[dayjaby]

Fixed.

[FooSoft]

A couple of other things for consistency with how other things work in Yomichan:

• Things on the Other tab should really be put onto Anki tab since they are related to Anki-specific functionality; Anki tab automatically gets disabled when running in standalone mode.
• The info icons don't do anything when not running as plugins and should therefore not show up when doing lookups.

[FooSoft]

The checkbox label also appears to be cut off:

[dayjaby]

Preferences window should be fine now. Overwrite icons are removed if running the standalone.

[dayjaby]

So what's the current status?

Atm I am writing some code that allows to "export vocabulary". Reading such a text results in having the necessary cards being automatically generated. That's practical if you want to share text with others.

Apart from that I started to work on Korean support as well. That's going to be in another pull request though ^^

[FooSoft]

It's a bunch changes, and I want to take the time to make sure I understand exactly how everything works before I merge. It's just taking me a little while cuz I have some stuff going on at school at the moment.

It will probably be another couple of weeks before I have a chance to really dig in ^^

Korean support sounds good, although it will be interesting trying to generalize the Japanese specific stuff scattered out throughout the program.

[dayjaby]

Yes, I think by adding another language, the Japanese stuff will get generalized out automatically.

[dayjaby]

As mentioned before, vocabulary is "exported" now. That means the entire definition of a vocabulary is placed into the text file now too.

Apart from that, upon opening a text file, any music/video/... file gets opened too, that has the same prefix. E.g. if I open AcchiKocchi1.txt, then AcchiKocchi1.mkv will be opened too. I did that, because most of my study materials that I use in Yomichan I got from some audio or video source. Having these available to accompany the reading is quite useful.

[dayjaby]

@FooSoft I've seen in your github activities that you started working on Yomichan again. I wondered if there is some common ground for collaboration, even though my previous commits deviate from Yomichan's original usage. Nonetheless, things like adding new languages, tiding up and modularizing previous code might be beneficial for both of us.

And I'd like to hear about your future plans :) If you'd like to discuss things in private, feel free to email me on dayjaby@gmail.com

[dayjaby]

Closed as it's improbable this is going to be merged in the near future.