Dictionary import very slow on Firefox Android

[starkgate]

Description Importing the jitendex dictionary on Firefox for Android is very slow, to the point where I was not sure if the process crashed or it's just taking a long time. I've started over twice, first it got stuck (as in, the progress bar didn't move for over 20mn) at step 3 of 6 at 78%. On the second try it got stuck at the same step but at 82%. After about 25mn it started progressing again.

Importing the same dictionary on a Windows laptop with Firefox was still quite slow (5mn), but it did finish successfully. Is this simply due to the difference in computing power or is there an actual bug?

Browser version Firefox Nightly for Android (124.0), Samsung S9 Firefox for Windows (122.0)

Yomitan version 23.12.29.0

[afn478]

What are your import times on chromium based browsers, e.g. chrome or brave on desktop and kiwi on android?

[starkgate]

What are your import times on chromium based browsers, e.g. chrome or brave on desktop and kiwi on android?

I would have to do an actual benchmark to answer you. On Chrome at least, the import time seemed similar to Firefox. I'm not sure if I want to put myself through the trouble of installing kiwi and waiting another 30mn to confirm if it is as slow on Android. If it's really needed, let me know.

[djahandarie]

Hmm... no clear answer, but to share what I know (which is entirely regarding desktop browsers, not mobile):

• Ever since we redid the schema validation using ajv, it's quite fast and only takes a few seconds
• Otherwise, the major time usage is just from doing a ton of IndexedDB operations
• In general, IndexedDB performs better on Firefox as it's backed by sqlite, there as opposed to LevelDB (like on Chrome)
• Presumably unzipping also takes up some CPU but it's probably not the major concern

I'm not sure how IndexedDB implementations work on Android browsers. But either way I'm not sure there's much we can do about it.

In terms of next steps, I think someone would need to spend time specifically importing and debugging the latest jitendex to see what part of the import is generating lots of IndexedDB operations (compared to other dicts). Of course it's also possible that there is some other logic in the import code that is misbehaving and doing something super-linear in terms of time complexity too.

[djahandarie]

(cc @stephenmk)

[starkgate]

Hmm... no clear answer, but to share what I know (which is entirely regarding desktop browsers, not mobile):

* Ever since we redid the schema validation using ajv, it's quite fast and only takes a few seconds

* Otherwise, the major time usage is just from doing a ton of IndexedDB operations

* In general, IndexedDB performs better on Firefox as it's backed by sqlite, there as opposed to LevelDB (like on Chrome)

* Presumably unzipping also takes up some CPU but it's probably not the major concern

I'm not sure how IndexedDB implementations work on Android browsers. But either way I'm not sure there's much we can do about it.

In terms of next steps, I think someone would need to spend time specifically importing and debugging the latest jitendex to see what part of the import is generating lots of IndexedDB operations (compared to other dicts). Of course it's also possible that there is some other logic in the import code that is misbehaving and doing something super-linear in terms of time complexity too.

Thanks for the detailed reply. For reference, the import has been running for ~1h20mn and is now at 28% of step 6 of 6. Unfortunately my experience with mobile is quite limited, but if you need any more information from me or help debugging Firefox on Android, let me know.

[stephenmk]

Firefox has always had terrible performance for me (PC, Linux). Import times take 10x as long as Chromium. Specifically "step 6 of 6" of the import process (importing the data after media has been loaded). I've been reluctant to raise an issue about it because I'm not sure if it's due to something peculiar with my PC.

I'd find it hard to believe that this is an issue specific to jitendex. I expect that any similarly large and complex dictionary would have the same problem.

[djahandarie]

I'd find it hard to believe that this is an issue specific to jitendex. I expect that any similarly large and complex dictionary would have the same problem.

Indeed, but I think we need to determine what dimension it is "complex" on that causes this problem. Because we have many other large dictionaries that (I believe) perform better, if solely comparing file size, such as https://github.com/MarvNC/wikipedia-yomitan (196 MB), or https://github.com/MarvNC/yomichan-dictionaries/raw/master/dl/%5BMonolingual%5D%20Pixiv.zip (88M), compared to jitindex (26M).

How do these other large dictionaries perform for you?

[stephenmk]

I got out my stopwatch and timed the imports. I guess that I should have built Yomitan from git instead of using the stable versions; I didn't realize the stable versions of Yomitan were different for each browser until after I had spent 30 minutes importing dictionaries into Firefox. In any case, I don't think that the newer version gave Chromium an advantage, because I think the older version on Chromium was equally as fast.

I started with a clean environment for each browser (new instance, new profile folder, no other extensions installed, etc.). I'm using an Intel i7-3770K processor. Newer hardware would probably be much faster.

Dictionary File	Dictionary Size	Browser	Extension Version	Step 6 time
[JA-JA Encyclopedia] JA Wikipedia [2022-12-01] (v1.4.0).zip	195.9 MiB	Firefox Browser Developer 123.0b3 (64-bit)	Yomitan 23.12.29.0	18m:50s (1130 seconds)
jitendex-yomitan-2024-01-28.zip	26.4 MiB	Firefox Browser Developer 123.0b3 (64-bit)	Yomitan 23.12.29.0	11m:55s (715 seconds)
[JA-JA Encyclopedia] JA Wikipedia [2022-12-01] (v1.4.0).zip	195.9 MiB	Chromium Version 121.0.6167.85	Yomitan 24.01.14.0	7m:52s (472 seconds)
jitendex-yomitan-2024-01-28.zip	26.4 MiB	Chromium Version 121.0.6167.85	Yomitan 24.01.14.0	2m:58s (178 seconds)

[starkgate]

Tested the import on another computer (i7-11700K), it took about 6mn with Firefox 122. I can only assume the reason it takes so long on mobile is hardware-bound, either RAM or CPU. In case it's useful, here are the CPU and IO usage plots during the import: