Open chrislongros opened 10 months ago
I'm very sorry to hear that! Thanks for reporting!
However, in my testing (on Linux with Anki 23.12.1 Qt6), manual snapshots (via File > CrowdAnki: Snapshot
) work, as do the automated snapshot on opening and closing Anki (when the relevant option is turned on).
(Edit (in case you read the previous version of this comment): snapshots on closing also seem to work — I got confused...)
Could you please:
Thanks and sorry again!
Anki 23.12.1 (1a1d4d54) (ao)
Python 3.9.15 Qt 6.6.1 PyQt 6.6.1
Platform: Linux-6.6.8-arch1-1-x86_64-with-glibc2.38
Traceback (most recent call last):
File "/home/chris/.local/share/Anki2/addons21/1788670778/dist/dulwich/file.py", line 148, in __init__
fd = os.open(
FileExistsError: [Errno 17] File exists: '/home/chris/.local/share/Anki2/addons21/1788670778/user_files/Chris/MyDeck__Medicine__a_YelloW_deck__The_ABSITE_Review__5th_edition__Chapter_15__Trauma/.git/objects/3c/9a31cef15582487e1dad1fde925062de89958a.lock'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "aqt.progress", line 118, in handler
File "aqt.main", line 218, in on_window_init
File "aqt.main", line 317, in setupProfile
File "aqt.main", line 496, in loadProfile
File "_aqt.hooks", line 4107, in __call__
File "anki.hooks", line 34, in runHook
File "/home/chris/.local/share/Anki2/addons21/1788670778/history/archiver_vendor.py", line 42, in snapshot_on_sync
self.do_snapshot('CrowdAnki: Snapshot on sync')
File "/home/chris/.local/share/Anki2/addons21/1788670778/history/archiver_vendor.py", line 50, in do_snapshot
self.all_deck_archiver().archive(overrides=self.overrides(),
File "/home/chris/.local/share/Anki2/addons21/1788670778/history/archiver.py", line 23, in archive
self.deck_archiver_supplier(deck).archive(reason=reason)
File "/home/chris/.local/share/Anki2/addons21/1788670778/history/anki_deck_archiver.py", line 22, in archive
repo.stage_all()
File "/home/chris/.local/share/Anki2/addons21/1788670778/history/dulwich_repo.py", line 32, in stage_all
self.dulwich_repo.stage(status.untracked + status.unstaged)
File "/home/chris/.local/share/Anki2/addons21/1788670778/dist/dulwich/repo.py", line 1374, in stage
self.object_store.add_object(blob)
File "/home/chris/.local/share/Anki2/addons21/1788670778/dist/dulwich/object_store.py", line 940, in add_object
with GitFile(path, "wb", mask=PACK_MODE) as f:
File "/home/chris/.local/share/Anki2/addons21/1788670778/dist/dulwich/file.py", line 92, in GitFile
return _GitFile(filename, mode, bufsize, mask)
File "/home/chris/.local/share/Anki2/addons21/1788670778/dist/dulwich/file.py", line 154, in __init__
raise FileLocked(filename, self._lockfilename) from exc
dulwich.file.FileLocked: ('/home/chris/.local/share/Anki2/addons21/1788670778/user_files/Chris/MyDeck__Medicine__a_YelloW_deck__The_ABSITE_Review__5th_edition__Chapter_15__Trauma/.git/objects/3c/9a31cef15582487e1dad1fde925062de89958a', '/home/chris/.local/share/Anki2/addons21/1788670778/user_files/Chris/MyDeck__Medicine__a_YelloW_deck__The_ABSITE_Review__5th_edition__Chapter_15__Trauma/.git/objects/3c/9a31cef15582487e1dad1fde925062de89958a.lock')
===Add-ons (active)===
(add-on provided name [Add-on folder, installed at, version, is config changed])
BetterSearch ['1052724801', 2023-06-15T22:44, 'None', '']
Button Colours Good Again ['2494384865', 2022-09-27T12:18, 'None', '']
CrowdAnki JSON exportimport Edit history Collaborate on deck creation ['1788670778', 2023-10-30T19:57, 'None', mod]
Edit Field During Review Cloze ['385888438', 2023-11-01T05:53, '6.17', mod]
FSRS4Anki Helper ['759844606', 2023-12-29T16:38, 'None', mod]
Highlight Search Results in the Browser ['225180905', 2023-10-21T21:17, 'None', '']
Hint Hotkeys ['1844908621', 2023-04-14T16:13, 'None', '']
More Overview Stats 21 ['738807903', 2021-07-30T21:13, 'None', '']
Review Heatmap ['1771074083', 2022-06-30T03:43, 'None', '']
===IDs of active AnkiWeb add-ons===
1052724801 1771074083 1788670778 1844908621 225180905 2494384865 385888438 738807903 759844606
===Add-ons (inactive)===
(add-on provided name [Add-on folder, installed at, version, is config changed])
AnkiHub ['1322529746', 2023-12-27T21:22, 'None', '']
The application just crashes unexpectedly. I managed to only export my data in a JSON file just once. After that when the add-on is enabled, Anki just crashes on start. Auto sync on open/close is enabled. Thanks for your quick response!
Oh no, this looks like a dulwich/git issue! :( (i.e. it's possible that it's indirectly caused by incompatibility between CrowdAnki and the newest version of Anki, but it might just be bad luck).
The direct issue is that at some point, when creating a snapshot, CrowdAnki (precisely the (otherwise amazing!) dulwich library which we use to handle git operations) did not remove a temporary git lock file (the lock files are there to make sure that all git operations are atomic). (For some as yet unknown reason — hopefully just a very rare timing issue, but possibly not.)
This lock file now persists and prevents CrowdAnki from doing anything with that git repository (because dulwich sees the lock file and believes that there is a stale/unfinished operation).
I believe that it should be safe to just remove the lock
file, but just in case it's probably better to move it somewhere — say try (in a terminal):
mv -i /home/chris/.local/share/Anki2/addons21/1788670778/user_files/Chris/MyDeck__Medicine__a_YelloW_deck__The_ABSITE_Review__5th_edition__Chapter_15__Trauma/.git/objects/3c/9a31cef15582487e1dad1fde925062de89958a.lock ~/temporary_lockfile
and then try opening Anki with CrowdAnki enabled. Hopefully this was a one-off issue and snapshots will continue as normal. (If not please complain here!)
Followed your instructions and removed the .lock file but another one appears. Keep in mind that my collection is relatively big and the JSON exported file is over 100 Mb, that git normally handles. Would git lfs be a better option? Is it combatible with CrowdAnki?
Anki 23.12.1 (8f77e519) (src) (ao)
Python 3.9.15 Qt 6.6.1 PyQt 6.6.1
Platform: Linux-6.6.8-arch1-1-x86_64-with-glibc2.38
Traceback (most recent call last):
File "/home/chris/.local/share/Anki2/addons21/1788670778/dist/dulwich/file.py", line 148, in __init__
fd = os.open(
FileExistsError: [Errno 17] File exists: '/home/chris/.local/share/Anki2/addons21/1788670778/user_files/Chris/MyDeck__Medicine__AnKing__Zanki_Step_Decks__Zanki_Neurology__Neuro_Misc/.git/objects/db/de378783376bef535b843dbb9b62c795f4084f.lock'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/chris/anki/ankidev/qt/aqt/progress.py", line 118, in handler
func()
File "/home/chris/anki/ankidev/qt/aqt/main.py", line 218, in on_window_init
fn()
File "/home/chris/anki/ankidev/qt/aqt/main.py", line 317, in setupProfile
self.loadProfile()
File "/home/chris/anki/ankidev/qt/aqt/main.py", line 496, in loadProfile
gui_hooks.profile_did_open()
File "/home/chris/anki/ankidev/out/qt/_aqt/hooks.py", line 4107, in __call__
anki.hooks.runHook("profileLoaded")
File "/home/chris/anki/ankidev/pylib/anki/hooks.py", line 34, in runHook
func(*args)
File "/home/chris/.local/share/Anki2/addons21/1788670778/history/archiver_vendor.py", line 42, in snapshot_on_sync
self.do_snapshot('CrowdAnki: Snapshot on sync')
File "/home/chris/.local/share/Anki2/addons21/1788670778/history/archiver_vendor.py", line 50, in do_snapshot
self.all_deck_archiver().archive(overrides=self.overrides(),
File "/home/chris/.local/share/Anki2/addons21/1788670778/history/archiver.py", line 23, in archive
self.deck_archiver_supplier(deck).archive(reason=reason)
File "/home/chris/.local/share/Anki2/addons21/1788670778/history/anki_deck_archiver.py", line 22, in archive
repo.stage_all()
File "/home/chris/.local/share/Anki2/addons21/1788670778/history/dulwich_repo.py", line 32, in stage_all
self.dulwich_repo.stage(status.untracked + status.unstaged)
File "/home/chris/.local/share/Anki2/addons21/1788670778/dist/dulwich/repo.py", line 1374, in stage
self.object_store.add_object(blob)
File "/home/chris/.local/share/Anki2/addons21/1788670778/dist/dulwich/object_store.py", line 940, in add_object
with GitFile(path, "wb", mask=PACK_MODE) as f:
File "/home/chris/.local/share/Anki2/addons21/1788670778/dist/dulwich/file.py", line 92, in GitFile
return _GitFile(filename, mode, bufsize, mask)
File "/home/chris/.local/share/Anki2/addons21/1788670778/dist/dulwich/file.py", line 154, in __init__
raise FileLocked(filename, self._lockfilename) from exc
dulwich.file.FileLocked: ('/home/chris/.local/share/Anki2/addons21/1788670778/user_files/Chris/MyDeck__Medicine__AnKing__Zanki_Step_Decks__Zanki_Neurology__Neuro_Misc/.git/objects/db/de378783376bef535b843dbb9b62c795f4084f', '/home/chris/.local/share/Anki2/addons21/1788670778/user_files/Chris/MyDeck__Medicine__AnKing__Zanki_Step_Decks__Zanki_Neurology__Neuro_Misc/.git/objects/db/de378783376bef535b843dbb9b62c795f4084f.lock')
===Add-ons (active)===
(add-on provided name [Add-on folder, installed at, version, is config changed])
BetterSearch ['1052724801', 2023-06-15T22:44, 'None', '']
Button Colours Good Again ['2494384865', 2022-09-27T12:18, 'None', '']
CrowdAnki JSON exportimport Edit history Collaborate on deck creation ['1788670778', 2023-10-30T19:57, 'None', mod]
Edit Field During Review Cloze ['385888438', 2023-11-01T05:53, '6.17', mod]
FSRS4Anki Helper ['759844606', 2023-12-29T16:38, 'None', mod]
Highlight Search Results in the Browser ['225180905', 2023-10-21T21:17, 'None', '']
Hint Hotkeys ['1844908621', 2023-04-14T16:13, 'None', '']
More Overview Stats 21 ['738807903', 2021-07-30T21:13, 'None', '']
Review Heatmap ['1771074083', 2022-06-30T03:43, 'None', '']
===IDs of active AnkiWeb add-ons===
1052724801 1771074083 1788670778 1844908621 225180905 2494384865 385888438 738807903 759844606
===Add-ons (inactive)===
(add-on provided name [Add-on folder, installed at, version, is config changed])
AnkiHub ['1322529746', 2023-12-27T21:22, 'None', '']
JSON exported file is over 100 Mb
I'll try to see if I can reproduce your issue with a large deck — if I can that will make debugging/testing smoother since I won't need to ask you to try things.
that git normally handles
Unfortunately, we don't directly use git, but instead use dulwich which has a ground-up pure-python implementation of git. (It has to be pure-python to be uploadable to AnkiWeb. We can't just "shell out" to git because most users won't have git installed.) If the issue turns out to be a not-easily-workaroundable performance issue in dulwich, then one solution might be to have an option (off-by-default) for CrowdAnki to directly call git (if the user has it installed). (Or possibly use libgit, but that's tricky because Anki doesn't access the "normal" python site packages.)
Would git lfs be a better option?
Possibly. Just allowing normal git to be called would be the first step, though. (To use git lfs we'd need to call the binary anyway, as dulwich doesn't fully support LFS (yet?).)
To keep you informed, I've tested with heavy decks (3 GB deck.json) and with large-ish amounts of media (10000 (small-ish) randomly generated images), and I'm still unable to reproduce the issue where the lock file is generated.
When I have more time I'll test some more situations (e.g. on Wayland), but for the time being I have no idea. (My best guess is that on your system Anki is (for some reason) killed before the creation of the snapshot is complete, but I have no idea why.)
If/when I have more time I'll also see if adding the option to use git directly might be straightforward and if it might help you.
As an alternative way of keeping a history of your deck you might, perhaps, in the meantime want to try ki. (Not affiliated and haven't actually tried it, but the creator had reached out to me suggesting it for another project, and it looks cool — its main disadvantage is that it's a separate tool.)
Thank you very much for your quick response and your time !! :)
I recently tried the add-on on Anki 23.12.1, but unfortunately although I could create a json file, snapshots do not work. Could you please update the add-on to be compatible with the latest Anki version? I find it very useful for version control of my changes. I would really be thankful.
Christos Longros