Updating of downloaded decks is too slow

[hssm]

Originally reported on Google Code with ID 228

Some shared downloaded decks easily take more than one hour for updating (eg Moder Chinese
Character Frequency List - 9937 facts).

During that time the phone is working very intensely in the background even after AnkiDroid
is closed, slowing down general performance.

While the PAUSE/RESUME capability will help fix that issue a bit, any idea for speeding
up the overall process is welcome.

Reported by inigo.aldana on 2010-09-23 20:28:53

[hssm]

Some very large decks simply do not finish - a Japanese deck I downloaded (~15k facts)
didn't complete in 10 hours of runtime and eventually the phone spontaneously rebooted
(from overheating?).

I think this updating process could be moved into a background thread that runs while
the user is running the deck. Say 100 cards could be updated when the deck is downloaded
and the rest could be updated while AnkiDroid is running normally.

Reported by Urzumph on 2010-11-02 08:55:31

[hssm]

Hi Urzumph,

Thanks for your feedback. Actually the application does the updating in the background,
so if you close Anki and do other things, the deck will still be getting updated.

BUT, the updating process is very intensive and you will see that it will slow down
your phone in general. Besides when it's doing it in the background it's obviously
even slower, as less resources are allocated.

The rebooting that you mention is quite alerting, I haven't seen that before. If it's
due to overheating, then we need to find a solution.

I have few ideas, but they are all quite laborious and I'm working on other things
at the moment:
- Can we use an external web app to do the updating? I considered App Engine, but it's
too unyielding for this kind of thing (lots of restrictions). Anyone know of an alternative
(doesn't have to be cloud based)?
- Copying the deck in app space (is it possible?) and doing the update there? It should
be faster than the SD-Card.
- Optimization of the updating process, although I don't expect miracles there, SQLite
for Android is simply slow.

On the up side, the libanki1.1 upgrade that I'm working on at the moment should have
a speed up effect even on this (due to getting rid of some indices).

I want to revisit this process when I finish the upgrade above, but in the meantime
I suggest you avoid downloading too big decks. Also if you don't mind telling us the
name of the deck, please do so, so we can use it for testing any speedup work.

Reported by inigo.aldana on 2010-11-02 13:26:14

[hssm]

Hi inigo.aldana,

By "in the background" I meant while the actual flashcards were running. Once we've
updated enough flashcards to complete a session, the user should be able to run the
flashcard set as normal (view cards, select hard/medium/easy, etc) while the update
process is running. In particular for large decks like the one I was using, it will
take a lot of runtime to finish all the flashcards, and I don't see any reason it needs
to all be done at once. So long as anki can update faster than I learn, doing the update
to all the cards while learning already updated cards should be no problem. I have
no idea if libanki supports that or not. I'm going to dive into the code to see what
I can do when I have a chance.

The deck I was using was "Monster Japanese Kanji + Vocab + Phrases" and although it
was ~15k facts, it was ~30k cards.

Reported by Urzumph on 2010-11-02 14:02:42

[hssm]

That's a good idea with potential. The only issue I see is whether Anki Desktop can
accept partially updated decks. If it can, then there's no reason we should update
the whole deck before we can start using it.

Kostas

Reported by inigo.aldana on 2010-11-02 14:20:13

[hssm]

That's a good idea with potential. The only issue I see is whether Anki Desktop can
accept partially updated decks. If it can, then there's no reason we should update
the whole deck before we can start using it.

Kostas

Reported by inigo.aldana on 2010-11-02 14:28:40

[hssm]

> The only issue I see is whether Anki Desktop can accept partially updated decks.

I am not sure how it does it because I haven't had time to look at the code, but in
fact, Anki Desktop updates the cards in decks like that (so, Anki Desktop does accept
partially updated and non-updated decks with no problem). I guess either when opening
the deck or probably before showing the card.

My main concern with this idea is how do we know "we've updated enough flashcards to
complete a session"? Should we have to update the cards in the same order they are
going to be presented to the user?

Reported by edu.zasu on 2010-11-02 15:06:25

[hssm]

Hi Edu,

Do we even need to have the cards updated, to start reviewing them? Can't we just do
it on the fly, ie generate the question/answer fields just prior to showing the card
to the user, then actually saving the updated fields when we save the new spacing,
after he/she has answered?
If I remember well, the updating process doesn't affect the scheduling of the cards
at all (unless I missed something).

Again that depends on Anki Desktop being able to handle such partially updated decks.
Also any search/browse functionality that we intent to add will probably not work properly
until the deck is fully updated.

I'd also like to mention that the new libanki1.1 pre-schedules the cards in a queue,
so we don't have to wonder how many cards we need for the next session.

Kostas

Reported by inigo.aldana on 2010-11-02 15:24:19

[hssm]

I've written some code for this:
https://github.com/urzumph/Anki-Android/commit/68d08b4daaca0f580e0cf6731d702126ef537a76

Still needs some work to clean the code up a bit to make it presentable, but it works,
albeit with limited testing.

Reported by Urzumph on 2010-11-08 12:41:17

[hssm]

Unfortunately I can't get eclipse to compile the Anki source without changes, and I
don't know enough about how git works to be able to make an easily-mergable patch.
Sorry. I've tested the code in the above commit a few times and it seems to work, so
if anyone with a working eclipse wants to try and merge my changes it shouldn't take
long.

Reported by Urzumph on 2010-11-21 02:11:58

[hssm]

Reported by nicolas.raoul on 2011-01-12 09:59:25

• Labels added: Priority-Low
• Labels removed: Priority-Medium

[hssm]

We can close this, there's no updating taking place anymore due to changes in the server.

Reported by inigo.aldana on 2011-03-06 16:11:27

[hssm]

Reported by nicolas.raoul on 2011-03-06 16:20:50

• Status changed: FixedInDev

[hssm]

Fixed in AnkiDroid 0.6, available on Google's Android Market.

Reported by nicolas.raoul on 2011-04-22 02:32:00

• Status changed: Fixed