Errors on Anki startup and SIAC document open: "embedded null byte"

brp-optics commented 2 years ago

With the release of 2.1.44 I upgraded, performing some sort of recommended deck upgrade, but SIAC didn't work at the time, so I downgraded again. The only other Add-On I have/had installed is "Auto Ease Factor."

Since downgrading, whenever Anki loads SIAC, I get the error reproduced below.

The 'Create New' window opens, but trying to actually add a new item causes the error to pop up. 'Read First in Queue' opens to the 'Add' window, but the part which should be SIAC is a mostly-grey box (see attached image:) read_first_in_queue 'Quick Open' opens the correct window, but attempting to read one of the documents has the same effect as 'Read First in Queue.' 'Settings' seems to work; but I haven't tried modifying them.

The error is persistent across uninstalling SIAC, reinstalling Anki, and reinstalling SIAC. It is persistent with other Add-Ons disabled. The error persists into a new (clean) Anki user profile. The error disappears if SIAC is uninstalled, but does not appear on a clean install of Anki when my other addon (Auto Ease Factor) is installed alone.

Error
An error occurred. Please start Anki while holding down the shift key, which will temporarily disable the add-ons you have installed.
If the issue only occurs when add-ons are enabled, please use the Tools > Add-ons menu item to disable some add-ons and restart Anki, repeating until you discover the add-on that is causing the problem.
When you've discovered the add-on that is causing the problem, please report the issue on the add-on support site.
Debug info:
Anki 2.1.40 (cf446733) Python 3.8.6 Qt 5.14.2 PyQt 5.14.2
Platform: Windows 7
Flags: frz=True ao=True sv=1
Add-ons, last update check: 2021-07-28 16:41:23

Caught exception:
Traceback (most recent call last):
  File "C:\...\1781298089\src\utility\date.py", line 56, in dt_from_stamp
    return datetime.strptime(stamp, '%Y-%m-%d-%H-%M-%S')
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\release\lib\site-packages\pyinstaller-4.0.dev0-py3.8.egg\PyInstaller\loader\pyimod03_importers.py", line 497, in exec_module
  File "_strptime.py", line 268, in <module>
  File "_strptime.py", line 182, in __init__
  File "_strptime.py", line 70, in __init__
  File "_strptime.py", line 83, in __calc_weekday
  File "_strptime.py", line 83, in <listcomp>
  File "calendar.py", line 82, in __getitem__
ValueError: embedded null byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\...\1781298089\src\menubar.py", line 96, in queue_picker
    show_queue_picker()
  File "C:\...\1781298089\src\api.py", line 67, in show_queue_picker
    dialog = QueuePicker(mw.app.activeWindow())
  File "C:\...\1781298089\src\dialogs\queue_picker.py", line 75, in __init__
    self.setup_ui()
  File "C:\...\1781298089\src\dialogs\queue_picker.py", line 90, in setup_ui
    self.queue_widget = QueueWidget(self)
  File "C:\...\1781298089\src\dialogs\queue_picker.py", line 743, in __init__
    self.fill_list(self.t_view_left, queue)
  File "C:\...\1781298089\src\dialogs\queue_picker.py", line 937, in fill_list
    sched       = utility.date.next_instance_of_schedule_verbose(n.reminder) if n.has_schedule() else "-"
  File "C:\...\1781298089\src\utility\date.py", line 90, in next_instance_of_schedule_verbose
    due_dt      = dt_from_stamp(due)
  File "C:\...\1781298089\src\utility\date.py", line 58, in dt_from_stamp
    return datetime.strptime(stamp, '%Y-%m-%d %H:%M:%S')
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\release\lib\site-packages\pyinstaller-4.0.dev0-py3.8.egg\PyInstaller\loader\pyimod03_importers.py", line 497, in exec_module
  File "_strptime.py", line 268, in <module>
  File "_strptime.py", line 182, in __init__
  File "_strptime.py", line 70, in __init__
  File "_strptime.py", line 83, in __calc_weekday
  File "_strptime.py", line 83, in <listcomp>
  File "calendar.py", line 82, in __getitem__
ValueError: embedded null byte

fonol commented 2 years ago

Hi, that's a weird one. You said the error persists after reinstalling the add-on. Did you also delete the siac-notes.db file? It might have been stored outside your add-on's folder, so uninstalling the add-on might not have affected it. I guess there could be some corrupted date string in your file, which is causing the error. You can find the path to the file under Settings & Info > Info > Path to note DB.

brp-optics commented 2 years ago

Thank you so much for the pointer!

You were right that the database was corrupted. Unfortunately I'm at a loss for how to get it back. It seems that the 'read' table is itself corrupted, but some rows in the 'notes' table are what cause the error in Anki. Below, I attach a single-row example DB which causes the error.

The full details of how I found this example follow below:

Replacing the database with even the oldest backup I have didn't resolve the issue.

Running "PRAGMA integrity_check" on a free SQLite Reader on the original (corrupted) database gives the following output:

*** in database main ***
On tree page 126 cell 113: Rowid 45 out of order
On tree page 126 cell 112: Rowid 45 out of order
On tree page 126 cell 111: Rowid 45 out of order
On tree page 126 cell 110: Rowid 45 out of order
On tree page 126 cell 109: Rowid 45 out of order
On tree page 126 cell 108: Rowid 45 out of order
On tree page 126 cell 107: Rowid 45 out of order
On tree page 126 cell 106: Rowid 45 out of order
On tree page 126 cell 104: Rowid 45 out of order
On tree page 126 cell 102: Rowid 49 out of order
On tree page 126 cell 100: Rowid 45 out of order
On tree page 126 cell 99: Rowid 45 out of order
On tree page 126 cell 98: Rowid 45 out of order
On tree page 126 cell 97: Rowid 45 out of order
On tree page 126 cell 96: Rowid 45 out of order
On tree page 126 cell 94: Rowid 45 out of order
On tree page 126 cell 92: Rowid 45 out of order
On tree page 126 cell 91: Rowid 45 out of order
On tree page 126 cell 90: Rowid 45 out of order
On tree page 126 cell 89: Rowid 45 out of order
On tree page 126 cell 88: Rowid 45 out of order
On tree page 126 cell 87: Rowid 45 out of order
On tree page 126 cell 86: Rowid 45 out of order
On tree page 126 cell 85: Rowid 45 out of order
On tree page 126 cell 84: Rowid 45 out of order
On tree page 126 cell 83: Rowid 45 out of order
On tree page 126 cell 82: Rowid 45 out of order
On tree page 126 cell 81: Rowid 45 out of order
On tree page 126 cell 80: Rowid 45 out of order
On tree page 126 cell 79: Rowid 45 out of order
On tree page 126 cell 78: Rowid 45 out of order
On tree page 126 cell 77: Rowid 45 out of order
On tree page 126 cell 76: Rowid 45 out of order
On tree page 126 cell 75: Rowid 45 out of order
On tree page 126 cell 74: Rowid 45 out of order
On tree page 126 cell 73: Rowid 45 out of order
On tree page 126 cell 72: Rowid 45 out of order
On tree page 126 cell 71: Rowid 45 out of order
On tree page 126 cell 70: Rowid 45 out of order
On tree page 126 cell 69: Rowid 45 out of order
On tree page 126 cell 68: Rowid 45 out of order
On tree page 126 cell 67: Rowid 45 out of order
On tree page 126 cell 66: Rowid 45 out of order
On tree page 126 cell 65: Rowid 45 out of order
On tree page 126 cell 64: Rowid 45 out of order
On tree page 126 cell 63: Rowid 45 out of order
On tree page 126 cell 62: Rowid 45 out of order
On tree page 126 cell 61: Rowid 45 out of order
On tree page 126 cell 60: Rowid 45 out of order
On tree page 126 cell 59: Rowid 45 out of order
On tree page 126 cell 58: Rowid 45 out of order
On tree page 126 cell 57: Rowid 45 out of order
On tree page 126 cell 56: Rowid 45 out of order
On tree page 126 cell 55: Rowid 45 out of order
On tree page 126 cell 54: Rowid 45 out of order
On tree page 126 cell 53: Rowid 45 out of order
On tree page 126 cell 52: Rowid 45 out of order
On tree page 126 cell 51: Rowid 45 out of order
On tree page 126 cell 50: Rowid 45 out of order
On tree page 126 cell 49: Rowid 45 out of order
On tree page 126 cell 48: Rowid 45 out of order
On tree page 126 cell 47: Rowid 45 out of order
On tree page 126 cell 46: Rowid 45 out of order
On tree page 126 cell 45: Rowid 45 out of order
On tree page 126 cell 44: Rowid 45 out of order
On tree page 126 cell 43: Rowid 45 out of order
On tree page 126 cell 42: Rowid 45 out of order
On tree page 126 cell 41: Rowid 45 out of order
On tree page 126 cell 40: Rowid 45 out of order
On tree page 126 cell 39: Rowid 45 out of order
On tree page 126 cell 38: Rowid 45 out of order
On tree page 126 cell 37: Rowid 45 out of order
On tree page 126 cell 36: Rowid 45 out of order
On tree page 126 cell 35: Rowid 45 out of order
On tree page 126 cell 34: Rowid 45 out of order
On tree page 126 cell 33: Rowid 45 out of order
On tree page 126 cell 32: Rowid 45 out of order
On tree page 126 cell 31: Rowid 45 out of order
On tree page 126 cell 30: Rowid 45 out of order
On tree page 126 cell 29: Rowid 45 out of order
On tree page 126 cell 28: Rowid 45 out of order
On tree page 126 cell 27: Rowid 45 out of order
On tree page 126 cell 26: Rowid 45 out of order
On tree page 126 cell 25: Rowid 45 out of order
On tree page 126 cell 24: Rowid 45 out of order
On tree page 126 cell 23: Rowid 45 out of order
On tree page 126 cell 22: Rowid 45 out of order
On tree page 126 cell 21: Rowid 45 out of order
On tree page 126 cell 20: Rowid 45 out of order
On tree page 126 cell 19: Rowid 45 out of order
On tree page 126 cell 18: Rowid 45 out of order
On tree page 126 cell 17: Rowid 45 out of order
On tree page 126 cell 16: Rowid 45 out of order
On tree page 126 cell 15: Rowid 45 out of order
On tree page 126 cell 14: Rowid 45 out of order
On tree page 126 cell 13: Rowid 45 out of order
On tree page 126 cell 12: Rowid 45 out of order
On tree page 126 cell 11: Rowid 45 out of order
On tree page 126 cell 10: Rowid 45 out of order
On tree page 126 cell 9: Rowid 45 out of order

Opening the db file in Emacs and stripping out everything that started with "2020" or "2021", I was able to identify three things that looked like corrupted dates:

2021-03-07-1^@^@^B ^B^A^B^\
2020-11-25-09-52-4^@^@
2021-04-0^[\204^^E^A^A^A3\377^]^F

That last one was so corrupted that python readline() borks on it.

Based on what data I could see in emacs hexl-mode, I set these to: 2021-03-07-19-20-12^@[nullbyte] 2020-11-25-09-52-40^@ 2021-04-03-15-11-51^F

The edited database then had the following errors:

*** in database main ***
On tree page 126 cell 114: Rowid 605 out of order
Multiple uses for byte 645 of page 126

database disk image is malformed

... And Anki was still complaining about embedded null bytes.

So I decided to start anew. I moved the edited siac-notes.db to a new location (siac_notes_for_export.db), and re-opened Anki/SIAC to create a clear siac-notes.db file. Attaching to the existing SQLite DB, I was able to copy over one table at a time, as follows:

INSERT INTO siac_notes.table_name SELECT * FROM table_name;
PRAGMA integrity_check(next_table_name)

It turns out that the "embedded null byte" error is caused by the contents of the "notes" table (none of which have the date strings above), but the table which fails the SQLite integrity check is the "read" table:

I then went through adding one row to the 'notes' table at a time, and seeing whether Anki complains about the null byte. This is taking a while. So far, I have found 22 rows (out of 125) that do not cause the null byte error and six that do. I can't see any pattern, except that all the rows I have found which don't cause problems have the 'position' value set to "NULL." Alas, choosing only rows with 'NULL' position is not enough to fix the error.

Anyway, I attach a single-note example db which causes the following error when Anki opens:

Error
An error occurred. Please start Anki while holding down the shift key, which will temporarily disable the add-ons you have installed.
If the issue only occurs when add-ons are enabled, please use the Tools > Add-ons menu item to disable some add-ons and restart Anki, repeating until you discover the add-on that is causing the problem.
When you've discovered the add-on that is causing the problem, please report the issue on the add-on support site.
Debug info:
Anki 2.1.40 (cf446733) Python 3.8.6 Qt 5.14.2 PyQt 5.14.2
Platform: Windows 7
Flags: frz=True ao=True sv=1
Add-ons, last update check: 2021-08-18 11:31:36

Caught exception:
Traceback (most recent call last):
  File "C:\Users\...\1781298089\src\notes.py", line 2158, in _dt_from_date_str
    return datetime.strptime(dtst, '%Y-%m-%d-%H-%M-%S')
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\release\lib\site-packages\pyinstaller-4.0.dev0-py3.8.egg\PyInstaller\loader\pyimod03_importers.py", line 497, in exec_module
  File "_strptime.py", line 268, in <module>
  File "_strptime.py", line 182, in __init__
  File "_strptime.py", line 70, in __init__
  File "_strptime.py", line 83, in __calc_weekday
  File "_strptime.py", line 83, in <listcomp>
  File "calendar.py", line 82, in __getitem__
ValueError: embedded null byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "aqt\progress.py", line 52, in handler
  File "aqt\main.py", line 138, in on_window_init
  File "aqt\main.py", line 186, in setupProfileAfterWebviewsLoaded
  File "aqt\main.py", line 227, in setupProfile
  File "aqt\main.py", line 406, in loadProfile
  File "aqt\hooks_gen.py", line 2140, in __call__
  File "C:\Users\...\1781298089\src\__init__.py", line 91, in <lambda>
    gui_hooks.profile_did_open.append(lambda : recalculate_priority_queue(True))
  File "C:\Users\...\1781298089\src\notes.py", line 510, in recalculate_priority_queue
    days_delta = max(0, (datetime.now() - _dt_from_date_str(last_prio_creation)).total_seconds() / 86400.0)
  File "C:\...\1781298089\src\notes.py", line 2160, in _dt_from_date_str
    return datetime.strptime(dtst, '%Y-%m-%d %H:%M:%S')
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\release\lib\site-packages\pyinstaller-4.0.dev0-py3.8.egg\PyInstaller\loader\pyimod03_importers.py", line 497, in exec_module
  File "_strptime.py", line 268, in <module>
  File "_strptime.py", line 182, in __init__
  File "_strptime.py", line 70, in __init__
  File "_strptime.py", line 83, in __calc_weekday
  File "_strptime.py", line 83, in <listcomp>
  File "calendar.py", line 82, in __getitem__
ValueError: embedded null byte

siac-notes_minimal_example.zip

brp-optics commented 2 years ago

A bit of an update. I discovered upon trying to recreate the DB from scratch in Anki that the error appears even in new databases.

I upgraded Anki to 2.1.44, uninstalled all add-ons, deleted the .anki-siac-addon-data folder, and reinstalled SIAC.

Upon adding a new note to SIAC, I get the same error (immediately upon clicking "Create and Keep open"):

Error
An error occurred. Please start Anki while holding down the shift key, which will temporarily disable the add-ons you have installed.
If the issue only occurs when add-ons are enabled, please use the Tools > Add-ons menu item to disable some add-ons and restart Anki, repeating until you discover the add-on that is causing the problem.
When you've discovered the add-on that is causing the problem, please report the issue on the add-on support site.
Debug info:
Anki 2.1.44 (b2b3275f) Python 3.8.6 Qt 5.14.2 PyQt 5.14.2
Platform: Windows 7
Flags: frz=True ao=True sv=2
Add-ons, last update check: 2021-08-19 12:26:36

Caught exception:
Traceback (most recent call last):
  File "C:\...\1781298089\src\notes.py", line 2158, in _dt_from_date_str
    return datetime.strptime(dtst, '%Y-%m-%d-%H-%M-%S')
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\release\lib\site-packages\pyinstaller-4.0.dev0-py3.8.egg\PyInstaller\loader\pyimod03_importers.py", line 497, in exec_module
  File "_strptime.py", line 268, in <module>
  File "_strptime.py", line 182, in __init__
  File "_strptime.py", line 70, in __init__
  File "_strptime.py", line 83, in __calc_weekday
  File "_strptime.py", line 83, in <listcomp>
  File "calendar.py", line 82, in __getitem__
ValueError: embedded null byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\...\1781298089\src\dialogs\editor.py", line 256, in on_create_and_keep_open_clicked
    success = self._create_note()
  File "C:\...\1781298089\src\dialogs\editor.py", line 306, in _create_note
    create_note(title, text, source, tags, None, specific_schedule, priority, author, url = url)
  File "C:\...\1781298089\src\notes.py", line 324, in create_note
    recalculate_priority_queue()
  File "C:\...\1781298089\src\notes.py", line 510, in recalculate_priority_queue
    days_delta = max(0, (datetime.now() - _dt_from_date_str(last_prio_creation)).total_seconds() / 86400.0)
  File "C:\...\1781298089\src\notes.py", line 2160, in _dt_from_date_str
    return datetime.strptime(dtst, '%Y-%m-%d %H:%M:%S')
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\release\lib\site-packages\pyinstaller-4.0.dev0-py3.8.egg\PyInstaller\loader\pyimod03_importers.py", line 497, in exec_module
  File "_strptime.py", line 268, in <module>
  File "_strptime.py", line 182, in __init__
  File "_strptime.py", line 70, in __init__
  File "_strptime.py", line 83, in __calc_weekday
  File "_strptime.py", line 83, in <listcomp>
  File "calendar.py", line 82, in __getitem__
ValueError: embedded null byte

The resultant siac-notes DB is attached. siac-notes.zip

I guess the next step is to start adding debugging lines to notes.py and try to figure out which datestring is causing trouble.

fonol / anki-search-inside-add-card

Errors on Anki startup and SIAC document open: "embedded null byte" #285