mathijsvdv / phrasify

Anki add-on that uses LLMs like ChatGPT to turn your vocabulary flashcards into fresh sentences on the fly and have conversations using your vocabulary.
MIT License
3 stars 0 forks source link

Performance enhancement: cache card generation #10

Open mathijsvdv opened 10 months ago

mathijsvdv commented 10 months ago

New generated cards need to be shown to a user upon reviewing a card. Since we need to make a request to OpenAI, this can induce significant latency when reviewing cards.

Filling the cache - triggers

We can use multiple strategies to fill the cache. I'll list the pros and cons:

Strategy Async needed Pros Cons
Upon startup Yes Can make generated cards while user is busy with other things. Cards may be generated that aren't reviewed
Upon card review No Easy to implement Have to wait the first time a card is reviewed.
Upon card creation Yes Can make generated cards as soon as they are known in the system Card may not be reviewed, costing money
Upon deck view Yes Can make generated cards specific to deck being viewed Cards may not be ready by the time the user starts reviewing. User may not actually review the deck, costing money.
Upon deck review Yes Can make generated cards specific to deck being reviewed Will have to wait for the first card of the deck to be generated

Given these pros and cons, I will consider implementing in the following order:

  1. [x] Upon card review: easy to implement, will already see some benefits.
  2. [ ] Upon startup: can keep all the generated cards up to date so we don't have a first-time latency.
    • Need to think about how many concurrent requests we can send to ChatGPT. Check the rate limiter.
  3. [ ] Upon card creation: can probably identify an event that we can react to with a hook
  4. [ ] Upon deck review: can probably identify an event that we can react to with a hook

Preventing card review blocking

Especially when filling the cache upon startup or any other event outside card review, we need any OpenAI calls to be made in the background. Here are two possible solutions to this:

Technology implementation of the cache

Generated cards need to be persisted to disk, because we may not be in the same Anki session the next time we review the card. Anki may have been closed and then reopened.

Persisting to disk means we need to be able to read the cards quickly. This means we can store them in a database. The database is of a size of the same order of magnitude as the cards which the client already has on their local system. To avoid slow network traffic, it makes sense to also store this table locally.

mathijsvdv commented 10 months ago

Just implemented caching upon card review using a JSON file:

Limitations we need to iron out:

Here are some possible solutions: