Import without "Update existing notes" doesn't work

[kpspu]

Ok, I have been using this addon for a while and now I am ready to create my first serious issue. First of all, thank you for your incredibly useful addon. It saves me time and makes my life easier.

Now about import issue. When I try to import cards with such settings as you see on screenshot:

I can't do that because app doing "Excluding already existing..." task endlessly (for my testing more than 8 hours for 311 words). It looks like this:

I don't think I can give you an error report for this issue because it isn't accompanied with any errors.

Actually I have error report in the beginning of importing process, but I don't think it is related to this issue. Because I have this error report sometimes several times when I import words with "Update existing notes" setting and it doesn't prevent app from successful import.

But I can give you this error report below. Again, I am not sure it is related to this issue.

Error An error occurred. Please start Anki while holding down the shift key, which will temporarily disable the add-ons you have installed. If the issue only occurs when add-ons are enabled, please use the Tools > Add-ons menu item to disable some add-ons and restart Anki, repeating until you discover the add-on that is causing the problem. When you've discovered the add-on that is causing the problem, please report the issue on the add-on support site. Debug info: Anki 2.1.26 (70784154) Python 3.8.0 Qt 5.14.1 PyQt 5.14.1 Platform: Windows 10 Flags: frz=True ao=True sv=1 Add-ons, last update check: 2020-05-13 13:21:34

Caught exception: Traceback (most recent call last): File "C:\Users\slavik\AppData\Roaming\Anki2\addons21\1411073333\gui.py", line 344, in download_words filtered = self.filter_words(words) if not self.checkBoxUpdateNotes.isChecked() else words File "C:\Users\slavik\AppData\Roaming\Anki2\addons21\1411073333\gui.py", line 365, in filter_words words = [word for word in words if not utils.is_duplicate(word.get('wordValue'))] File "C:\Users\slavik\AppData\Roaming\Anki2\addons21\1411073333\gui.py", line 365, in words = [word for word in words if not utils.is_duplicate(word.get('wordValue'))] File "C:\Users\slavik\AppData\Roaming\Anki2\addons21\1411073333\utils.py", line 200, in is_duplicate return True if get_duplicates(word_value) else False File "C:\Users\slavik\AppData\Roaming\Anki2\addons21\1411073333\utils.py", line 188, in get_duplicates note_dupes = list(set(note_dupes1 + note_dupes2)) TypeError: unsupported operand type(s) for +: 'RepeatedScalarFieldContainer' and 'RepeatedScalarFieldContainer'

Can you please fix import without "update existing notes" feature? I need it very much, I have word and phrases what doesn't copied in Anki by some reason and I want to repeat import without affecting existing notes where I have done a lot of adjustments.

[vi3itor]

Hi! Thanks for kind words, I'm glad to hear that you find it useful. I will look into the issue and get back with a solution.

I remember reading a long time ago, that Anki didn't handle well big databases, but I doubt it's the case here. Damien has improved the client a lot since then, so it should work flawlessly with your dictionary. More likely, something has recently changed in the database API, so it takes ages to query the database somewhat 600 times (to check for words with and without apostrophes). And the error above gives some clues.

But I don't have a large enough deck to test the performance. It will be a great help if you can share your deck (without media and scheduling information). To do so click the gear icon next to LinguaLeo deck and choose Export:

untick media and scheduling info:

and send to 4yourquestions@gmail.com If you decide to send me the desk, I won't share it with anyone.

[vi3itor]

Oh, I just tested the latest version of Anki and it is very slow even with my deck of about 7000 words.

[kpspu]

Oh, I just tested the latest version of Anki and it is very slow even with my deck of about 7000 words.

In that case do you still need my Deck? It would be another deck, "English", my "LinguaLeo" deck is empty, I use it only for import.

I just have one idea. Might it be because my Dictionary contains lots of phrases? Do you have 7000 words or do you have 7000 words and phrases?

[vi3itor]

I fixed the issue but didn't update the add-on yet. I found another problem: Anki adds duplicates now (instead of the old behavior where it skipped them). I will investigate further. If you share your English deck, I'll test future versions of the add-on on it as well.

[vi3itor]

I fixed the problem with duplicate creation and uploaded a new version. Please check it out. It should work fast and even don't download media if the file already exists.

The problem appeared after Anki introduced a new search mechanism in version 2.1.24. Unfortunately, it means that you have to check your deck for duplicates and delete "extra" notes :(

Anki provides a mechanism to find duplicates and even tag them as 'duplicate', but you have to delete notes manually. Anki creates backups too, so it might be easier to restore a collection from earlier. To find duplicates go to the Browse window and in the menu select "Notes" -> "Find duplicates", choose to search in the field "en". When found, click "Tag duplicates". Now you can go to the Browse window and use filter to choose duplicated notes: tag:duplicate card:1. card:1 will show only 'en->ru' cards, but if you delete a note 'ru->en' card will be removed too.

[kpspu]

OMG, I have finished cleaning my deck just now. There were 7500 duplicated cards. I was lucky that I could delete 6550 of them by finding a distinguishing tag. But last 950... It was monotonous. But I did it :) After doing that, I checked your addon, and it works like you said: fast, without mistakes or creating duplicates. Thank you for your help! I close this issue. It is solved.