Missing Bookmarks

[jalcine]

Not sure where or how this is happening, but I run into this error when I attempt to sync my bookmarks:

calibre, version 6.13.0
ERROR: Unhandled exception: <b>AttributeError</b>:'list' object has no attribute 'values'

calibre 6.13  embedded-python: False
Linux-6.1.0-11-amd64-x86_64-with-glibc2.36 Linux ('64bit', 'ELF')
('Linux', '6.1.0-11-amd64', '#1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4 (2023-08-08)')
Python 3.11.2
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (2, 5, 0) && Package KFX (from KFX Input) (2, 5, 0) && ACE (1, 1, 6) && Annotations (1, 17, 13) && Audio M3U (1, 0, 0) && AudioBook_Duration (1, 0, 9) && Audit Log (1, 0, 19) && Barnes & Noble (1, 5, 2) && Check Books (0, 1, 8) && Count Pages (1, 13, 2) && Embed Comic Metadata (1, 6, 6) && FanFicFare (4, 27, 0) && Fantastic Fiction (1, 6, 4) && Find Duplicates (1, 10, 8) && Goodreads (1, 7, 9) && Goodreads Sync (1, 16, 3) && Goodreads_Rating (4, 0, 0) && Import List (1, 9, 1) && KFX metadata reader (from KFX Input) (2, 5, 0) && KFX Input (2, 5, 0) && KOReader Sync (0, 5, 2) && Kobo Books (1, 9, 2) && Kobo Utilities (2, 16, 10) && KoboTouchExtended (3, 6, 6) && Library Codes (1, 0, 65) && LibraryThing Match (0, 2, 2) && OverDrive Libby (0, 1, 9) && Overdrive Link (2, 56, 0) && Prettify Cover (1, 6, 0) && Quality Check (1, 13, 6) && Reading List (1, 15, 1) && Search The Internet (1, 10, 1) && Similar Stories (1, 0, 58) && Standard Ebooks (1, 0, 0) && Wiki Reader (2, 2, 1) && WordDumb (3, 29, 6) && ePub Extended Metadata {Writer} (0, 10, 2) && ePub Extended Metadata {Reader} (0, 10, 2) && ePub Extended Metadata (0, 10, 2) && Overdrive Link Metadata Source (2, 56, 0)
Traceback (most recent call last):
  File "calibre_plugins.koreader.action", line 707, in sync_to_calibre
    sidecar_contents = self.get_sidecar(device, sidecar_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "calibre_plugins.koreader.action", line 289, in get_sidecar
    parsed_contents = self.parse_sidecar_lua(decoded_contents)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "calibre_plugins.koreader.action", line 326, in parse_sidecar_lua
    for bookmark in decoded_lua['bookmarks'].values()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'values'

I can assume that there's some kind of mismatch happening here https://github.com/harmtemolder/koreader-calibre-plugin/blob/main/action.py#L319-L327 (is it possible that it's getting back a null value but the key itself exists).

[kyxap]

This is fixed with #23

[jalcine]

This hasn't been; I've upgraded manually from the tarball and I see a similar issue appear.

calibre 7.15  embedded-python: True
Linux-6.1.0-23-amd64-x86_64-with-glibc2.36 Linux ('64bit', 'ELF')
('Linux', '6.1.0-23-amd64', '#1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15)')
Python 3.11.5
Interface language: None
EXE path: /opt/calibre/bin/calibre
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (2, 15, 0) && Package KFX (from KFX Input) (2, 15, 0) && ACE (1, 1, 6) && Annotations (1, 17, 13) && Audio M3U (1, 0, 0) && AudioBook_Duration (1, 0, 9) && Audit Log (1, 0, 19) && Barnes & Noble (1, 5, 5) && Check Books (0, 1, 8) && Count Pages (1, 13, 6) && Embed Comic Metadata (1, 6, 6) && FanFicFare (4, 36, 0) && Fantastic Fiction (1, 7, 0) && Find Duplicates (1, 10, 9) && Goodreads (1, 8, 2) && Goodreads Sync (1, 16, 8) && Import List (1, 9, 4) && KFX metadata reader (from KFX Input) (2, 15, 0) && From KFX (2, 15, 0) && KFX Input (2, 15, 0) && KOReader Sync (0, 6, 3) && Kobo Books (1, 9, 2) && Kobo Utilities (2, 17, 1) && KoboTouchExtended (3, 6, 7) && LibraryThing Match (0, 2, 5) && OverDrive Libby (0, 1, 9) && Overdrive Link (2, 57, 0) && Prettify Cover (1, 6, 0) && Quality Check (1, 13, 12) && Reading List (1, 15, 4) && Search The Internet (1, 11, 2) && Similar Stories (1, 0, 58) && Standard Ebooks (1, 0, 0) && Wiki Reader (2, 2, 1) && WordDumb (3, 32, 1) && ePub Extended Metadata {Writer} (0, 11, 2) && ePub Extended Metadata {Reader} (0, 11, 2) && ePub Extended Metadata (0, 11, 2) && Overdrive Link Metadata Source (2, 57, 0)
Traceback (most recent call last):
  File "calibre_plugins.koreader.action", line 775, in sync_to_calibre
    sidecar_contents = self.get_sidecar(device, sidecar_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "calibre_plugins.koreader.action", line 367, in get_sidecar
    parsed_contents = parse_sidecar_lua(decoded_contents)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "calibre_plugins.koreader.action", line 127, in parse_sidecar_lua
    for bookmark in decoded_lua['bookmarks'].values()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'values'

There might be a few places where this is happening, could be worth introducing some sort of typing!

[jalcine]

It sucks that I can't figure out which file this is causing a problem with, I'd just provide it as a test case (unless you got an idea)!

[kyxap]

Please provide

Device:
Kobo ver:
OS: 

Also run calibre in debug mode calibre-debug -g and provide output.

With debug mode you can find which book(lua sidecar file) having the issue. Wound be also useful to get lua file to check the format and reproduce the issue.

[jalcine]

Here's the file.

Device: Kobo Clara 2E Kobo ver: 4.37.21586 OS: Kobo?

[jalcine]

Hm, I also see this happening in another book, info at metadata.epub.lua.txt. There's a chance that I'm getting some sort of a mismatch on the expected data format.

[kyxap]

forgot to ask what is your koreader version? they have updated lua file format since couple version so you may need to update to latest

[kyxap]

yeah, just checked with my lua files, your seems to be on old version, because this:

return {
    ["bookmarks"] = {

replaced with

return {
    ["annotations"] = {

[jalcine]

I'm on the latest version according to https://ota.koreader.rocks/, v2024.07. I think there's a chance the information didn't get migrated over somehow. So what do you suggest I do; replace that key?

[kyxap]

Do this:

• Backup <book_name>.sdr folder
• Delete <book_name>.sdr folder from device
• Open book, add new notes
• Try to sync again and see if issue gone
• Double check if format has annotations instead of bookmarks

and let me know

[kyxap]

for reference https://github.com/koreader/koreader/discussions/11929#discussioncomment-9612272

[kyxap]

btw as alternative you can try to add new bookmark and I assume it may trigger lua script update but I can't guarantee and who know it may break your notes so backup is preferable

[kyxap]

going to close since no response from the author, let me know if you still have this issue

[TheOnlyWayUp]

Hi! I'm facing this issue.

Edit: I was on KOReader v2023.10, updated to v2024.07

Still happens, I updated the plugin from v0.5.2 to v0.6.7

No dice, while updating Calibre itself, I took a look at the sdr folder. There's metadata.epub.lua, and metadata.epub.lua.old. Both of them were last modified when I finished reading, and haven't been updated with KOReader's update.

Edit: I opened the same book and highlighted a word, this updated the sdr file. (only for that book)

NEW:

        [1] = {
            ["chapter"] = "xxx",
            ["datetime"] = "2024-02-07 09:59:52",
            ["drawer"] = "lighten",
            ["page"] = "/body/DocFragment[2]/body/p[57]/text().0",
            ["pageno"] = 10,
            ["pos0"] = "/body/DocFragment[2]/body/p[57]/text().0",
            ["pos1"] = "/body/DocFragment[2]/body/p[57]/text().662",
            ["text"] = "xxx",
        }

OLD:

        [1] = {
            ["chapter"] = "xxx",
            ["datetime"] = "2024-01-07 01:05:52",
            ["highlighted"] = true,
            ["notes"] = "xxx",
            ["page"] = "/body/DocFragment[6]/body/p[352]/text().104",
            ["pos0"] = "/body/DocFragment[6]/body/p[352]/text().104",
            ["pos1"] = "/body/DocFragment[6]/body/p[352]/text().1183",

Mainly, the addition of a pageno, renaming of notes to text, and the removal of the highlighted key.

---

Edit: Ookay, I played around a bit. Turns out some books have weirdly formatted SDRs,

    {
        "bookmarks": [
            {
                "chapter": "",
                "datetime": "2023-11-02 14:52:29",
                "highlighted": True,
                "page": "/html/body/p[1469]/text().59",
                "pos0": "/html/body/p[1469]/text().59",
                "pos1": "/html/body/p[1469]/text().73",
            },

for example (decoded_lua ^)

LFG!

fix:

    if 'bookmarks' in decoded_lua:
        debug_print('calculating first and last bookmark dates')

        # ! here
        if type(decoded_lua['bookmarks']) is list:  
            decoded_lua['bookmarks'] = {i: bookmark for i, bookmark in enumerate(decoded_lua['bookmarks'])}

        bookmark_dates = [
            datetime.strptime(
                bookmark['datetime'],
                '%Y-%m-%d %H:%M:%S'
            ).replace(tzinfo=utc_tz)
            for bookmark in decoded_lua['bookmarks'].values()
        ]

        if len(bookmark_dates) > 0:
            decoded_lua['calculated'] = {
                'first_bookmark': min(bookmark_dates),
                'last_bookmark': max(bookmark_dates),
            }

    # ! here
    if 'annotations' in decoded_lua:
        debug_print('calculating first and last bookmark dates')
        bookmark_dates = [
            datetime.strptime(
                bookmark['datetime'],
                '%Y-%m-%d %H:%M:%S'
            ).replace(tzinfo=utc_tz)
            for bookmark in decoded_lua['annotations'].values()
        ]

        if len(bookmark_dates) > 0:
            decoded_lua['calculated'] = {
                'first_bookmark': min(bookmark_dates),
                'last_bookmark': max(bookmark_dates),
            }

It's mainly

        if type(decoded_lua['bookmarks']) is list:  
            decoded_lua['bookmarks'] = {i: bookmark for i, bookmark in enumerate(decoded_lua['bookmarks'])}

Which converts decoded_lua['bookmarks'] from an array of just highlights to a dict of idx's to their highlights. Ideally, we'd do i+1 as KOReader starts counting from 1. (Inplace, so any downstream reads to bookmarks will be in the same format)

I also duplicated the bookmarks block and replaced 'bookmarks' with 'annotations'. No errors (mixed library, some old, some in the weird format, and some new), I'm glad :)

Edit: Okay, after a bit more digging around, turns out the new sidecar format contains both bookmarks and annotations, making it backwards-compatible. The block for annotations is redundant.

Should I make the inplace update to decoded_lua['bookmarks'] a PR?

[TheOnlyWayUp]

cc @jalcine @kyxap

[kyxap]

@TheOnlyWayUp please make sure you using latest version of the plugin since 'bookmarks' with 'annotations' is fixed long time ago #23 if you still have issue open new one and provide required information for debugging or you can submit the PR, thanks