Suggestion: Merge highlights also if one of the book version has no highlights at all.

[richo67]

I often read a book on one device and take the second with me on my travels. I would like to copy highlights from the first to the second. If I open book on second device I have to make fake highlight to be able to merge it with highlights on other device.

[noembryo]

I think that this isn't very difficult to implement.. :o)

[noembryo]

I erased the highlights from a metadata file. Then I opened it again and tried to sync it with another version that had highlights. It did synced OK. The same happens if I archive a book with highlights and then open a version without and sync it with the archived version.

There is something I don't get here. What exactly is happening to your two versions? Can you attach any samples?

[richo67]

I have same book on two devices. I open book on both but only made highlight on one. When I scan two different folders, containing these folders of the same book, One that has highlights has green icon can not be merged with the other which has no green icon. See picture please. The Merge tool in Toolbar is disabled.

I am also attaching these two .lua files

luas.zip

[noembryo]

Thanks for the samples. I'll check them out...

[noembryo]

Well, the problem with the luas is that they created for different books. Maybe you re-exported your book from Calibre or edited it before opening it. The id of the book is computed the first time you opening it with KOReader.

[richo67]

interesting is that the book the epub files are binary same see picture with checksum. Could thi be because book was open with earlier version of koreader and then converted on one device and on the other device it was already open with the new version first time?

[noembryo]

Hmm, I don't know the way that computation is made but yes, that could be the case.. :o(

[richo67]

Which is the tag name of the id? I just deleted the sdr folder and reopen the book with same koreader, still can't merge.

[noembryo]

The partial_md5_checksum

[richo67]

that one is different, that is really crazy why? epub is exactly the same

[richo67]

I have delete it on kindle and reopen again now md5 is same, but I have no higlights now :-( when book is converted between different format basedon koreader version, it should also update the MD5 id, don't you think ?

[noembryo]

Forgot to mention that there is another MD5 id somewhere there, but I don't use it.. Try to copy this, too and see..

[noembryo]

You're right about the conversion, but I'm sure there must be a reason behind it. I remember there was an issue at the KOReader's github about all this, but I can't find it now. I'll look again later...

[noembryo]

I have delete it on kindle and reopen again now md5 is same, but I have no higlights now :-(

You can keep the metadata file that you send me and merge it using the newer version of KOHighlights.. ;o)

[richo67]

You can keep the metadata file that you send me and merge it using the newer version of KOHighlights.. ;o)

No I can't because the new koreader (same on both devices now) produced now the new MD5.

Maybe the MD5 calculation is taking into account new version of a DOM or some other new part without taking into account the epub. Maybe that is why MD5 changes when koreader changes the DOM or another internal rendering version. It will upgrade the rendering of the document but will not upgrade the MD5 from previous time. This I think may be a bug. As the new version of Koreader upgraded all but kept old MD5s.

P.S.: I just replace MD5 in old metadata (after the koreader said it must update something in book) with MD5 from new metadata (when book was directly, first time, open by new version of Koreader) and it is all working fine. I have highlights and all.

[noembryo]

P.S.: I just replace MD5 in old metadata (after the koreader said it must update something in book) with MD5 from new metadata (when book was directly, first time, open by new version of Koreader) and it is all working fine. I have highlights and all.

Great!

[noembryo]

So are we concluded that we don't need the "fake" highlights to make two versions of a book syncable?

[richo67]

So are we concluded that we don't need the "fake" highlights to make two versions of a book syncable?

I have not tested, but if you are certain that it work with empty, than it is fine. The book of course needs to be at least once opened, the sdr folder and metadata file must exists, correct?

Are you going to calculate MD5 yourself as id of a book or you going to use one of the MD5 in metadata file?

[noembryo]

I will use the partial_md5_checksum if there is one (and that is not always the case), or else I will use the `stats["MD5"] if there is one, or else I will calculate one myself (if the book file is available). A lot of checking.. :o)

[richo67]

Because of the MD5 errors in Koreader wouldn't be much better to calculate this on the fly to see if the books are the same? Then you 100% sure.

[noembryo]

Well, the bug was fixed, and I don't have the book available for all cases (e.g. in android, or if you copy the folders over to a PC). Its also needs more processing, so when scanning a lot of files it might slow down things a little..

[richo67]

Wow I never would dare to part the book and its sdr folder. Why would you do it?

Bug was fixed, but but MD5 was not recalculated. I also understand why same book was not synchronizing between devices. It would be great (and I know this is not really optimal to fix the issue with KOreader here), that in situation when you find out that book is not the same but otherwise like to be same (does KoHighlights use also other data to compare the books? author, title?) maybe you can fix the MD5 in lua file too, if it would be same, after calculations on the book's files.

[noembryo]

Wow I never would dare to part the book and its sdr folder. Why would you do it?

For my case, most of my highlighting is for book corrections, so once in a while that I connect to the PC I copy the newer sdr folders over, to fix the relevant books whenever I find time. This is an old habit (before the database) that is now maybe deprecated, but hey, habits die last..

Bug was fixed, but but MD5 was not recalculated. I also understand why same book was not synchronizing between devices. It would be great (and I know this is not really optimal to fix the issue with KOreader here), that in situation when you find out that book is not the same but otherwise like to be same (does KoHighlights use also other data to compare the books? author, title?) maybe you can fix the MD5 in lua file too, if it would be same, after calculations on the book's files.

Since its fixed, it seems to me as an overkill to implement a feature that is needed only for the books that affected by this bug. I don't compare other metadata like Author or Title, because they can be the same in totally different books. Also keep in mind that these comparisons are made with every selection change, so we have to take into account that we will slow down all the selections for something that is needed in an edge case scenario..

Hmm, what I could do though, is to have an option to just re-calculate the MD5s for the selected books.. But even this looks like an overkill (for such rare cases). Maybe if I could hide it somewhere..

[richo67]

Maybe I will learn to do some python and isolate some code from yours and made command line utility to recalculate the MD5 ;-)

[richo67]

I promise, this to be last suggestion in this month :-) On the right click popup menu on the book, would it be possible to add open the sdr folder menu item? This could be related to cleaning of additional metadata files after merge. Maybe also menu item recalculate MD5 ? - maybe good place to hide it?

[noembryo]

Keep in mind that the reason I started coding, was an obnoxious developer that had created a utility I used I lot, but keep on making me to jump over many hoops to accomplish my goals.. ;o)

Just added an action in right click menu of a book to recalculate the MD5 (if the book file is available of course).

[noembryo]

Oups, just saw your last comment, and the second part is already made.

For the first part I was thinking that maybe it would be better if double clicking on the path of the book's lua opened the sdr folder?

[richo67]

For the first part I was thinking that maybe it would be better if double clicking on the path of the book's lua opened the sdr folder?

That is also good option, not very visible though.

[richo67]

Keep in mind that the reason I started coding, was an obnoxious developer that had created a utility I used I lot, but keep on making me to jump over many hoops to accomplish my goals.. ;o)

:-) very valid reason.

[noembryo]

That is also good option, not very visible though.

You're right, but I just try not to clutter the right click menu with actions. Visibility counts though, so I have to re-arrange things a little..

[noembryo]

Added with v.1.3.0.0