Enhancement request - sync jobs and other features

[edo-jan]

The following is inteface / feature ideas you may consider for future development. The proposed interface is inspired by an old nifty program called "allway sync". The main feature allows creating sync jobs for each book. Each job resides in its own tab where it can be configured and run.

The second file show the sync direction configurator. This is how it is implemented in allway sync. Extremely flexible since one can select the sync direction, participating devices, as well as weather or not to propogate changes, deletions, or both.

The "Analyze" button produces the summary of changes for each device and the "Synchoronize" button - executes the job.

I hope this is in line with the direction you are taking your great little program. Thanks again for your service to the community!

The following shows some of the sync configuration options that one can change upon clicking on the "change" button:

[noembryo]

Thank you for your ideas and suggestions. Unfortunately, syncing highlights was not in the original plan for this app. It was made just to read the highlights on the PC. Syncing appeared as an afterthought. But since it exists, these ideas might help me to make it more user friendly.

The main difference of KOHighlights and this sync app (which obviously is created mainly for syncing), is that KOHighlights is totally device-agnostic. It can compare between, and work with, only two data parts: What is loaded from a directory, and a database. Sure, I can save some preset paths for the mount points of different devices, but I can have only one list of books at a time. And also only one database. Hmm.. I have to think more about it.. If you have any more ideas, add them to the pile.. 😉

[edo-jan]

If syncing appeared as an afterthought, it was a great afterthought indeed ! :) I could be wrong but imho the difference between two way sync and three way (or more) sync is doing it in sequence. Once the first pair finishes, the second pair starts, etc.

[noembryo]

I could be wrong but imho the difference between two way sync and three way (or more) sync is doing it in sequence. Once the first pair finishes, the second pair starts, etc.

The difference for me is to find the different elements to sync (the second pair, the third pair..) I have two lists of books as I wrote. The file list and the database list. Those pairs are books that exist in deferent file lists or databases. The problem I'm thinking is how to create a GUI way to let the user select all those different destinations. Without recreating half of the application again.. 😉 Don't worry, I'll think of something..

[noembryo]

The problem I'm thinking is how to create a GUI way to let the user select all those different destinations. Without recreating half of the application again.. 😉

Well, I didn't succeed.. I almost rewrote the app. This enhancement coincided with a major change in the KOReader's metadata format (not yet on current KOReader) and a major addition of GUI changes (dark theme and icons) So, this is what I came up with: Not finished yet, so if you have any suggestions..

[edo-jan]

Wow, this looks like a total re-write! I like the linear way you present the information and the dark theme looks cool. Correct me if I am wrong, it appears that your "sync groups" are the same as "jobs", right? If so does it mean that they are going to be persistent or must be re-created every time I open Kohighlights?

RE: the new menu icons. They look great. At the same time when I put on a hat of a new user (someone who has never used Kohighlights before), I would welcome more descriptive naming (or additional hint on mouse over). I would appreciate more info on what the buttons do exactly.

A) E.g. The "power" button on the left - am I correct to assume that it toggles the option to include the given group into the Group Sync? B) What are the "select" / "plus" / "minus" buttons? C) What would happen if the sync is attempted while neither "sync position", "merge highlights" or "update to archive" is checked? D) what would happen if the sync is attempted while some of the devices / sync paths for the given group are not connected / accessible (assuming the groups are persistent)?

Ideally, on the program-level menu (the main menu tool bar), I would expect to see the buttons that modify the groups - add, remove or sync (selected) groups.

The rest of the buttons that modify inside the groups should reside on the group- level sub-menus (currently this sub-menu has a single button called "Sync this group" as well as three check boxes) . I would find it more intuitive should these sub-menus had the "plus" and "minus" buttons allowing additions or removals of the existing (selected) sync paths respectively.

On a different note, there is a potential risk inherent in all sync programs. I am referring to a scenario where one of the sync sources accidently gets deleted and the deletions are then propagated to all destinations resulting in a total data loss. I understand to this end you have the "archive" feature. However, since currently archiving is a manual process I wonder if you would consider turning the archiving on by default. Ideally, the archive should be viewed as the historic version of the current dataset existing for each sync group on the "server". That means that Kohighlights database would be included in every sync group by default as a server copy. In case of accidental data corruption one would be able to revert to the last good copy.

In the future a cloud version of the program would make a good sense whereas the user would create a server copy as the very first step by synching his device to the server. Just an idea. Thanks for your great work! I know this can be a lonely jorney. Let me know if you need any help with testing.

[noembryo]

Wow, this looks like a total re-write! I like the linear way you present the information and the dark theme looks cool. Correct me if I am wrong, it appears that your "sync groups" are the same as "jobs", right? If so does it mean that they are going to be persistent or must be re-created every time I open Kohighlights?

They are persistent. Any change of any group is saved immediately,

RE: the new menu icons. They look great. At the same time when I put on a hat of a new user (someone who has never used Kohighlights before), I would welcome more descriptive naming (or additional hint on mouse over). I would appreciate more info on what the buttons do exactly.

The new GUI has the option to select between the old bitmap (colored) icons or the minimalistic ones (menu bottom-right) and of course every button has a tooltip.. 😛

A) E.g. The "power" button on the left - am I correct to assume that it toggles the option to include the given group into the Group Sync?

This button enables/disables this Sync group. A Sync group consists of various Sync paths. This let you sync all the enabled Sync groups with one click of the main Merge/Sync toolbar button. You can also sync the individual Sync groups one at a time, by pressing their "Sync this group" button.

B) What are the "select" / "plus" / "minus" buttons?

The select button lets you select the Sync path's address by browsing you drive. The plus button adds a new Sync path to the group and the minus removes the Sync path from the group.

C) What would happen if the sync is attempted while neither "sync position", "merge highlights" or "update to archive" is checked?

A popup will inform you that there is nothing to sync.

D) what would happen if the sync is attempted while some of the devices / sync paths for the given group are not connected / accessible (assuming the groups are persistent)?

When a sync path is not accessible or the metadata don't match (with the first) or has one of various problems that is checked against, it will be marked red and will not take part in the syncing process.

Ideally, on the program-level menu (the main menu tool bar), I would expect to see the buttons that modify the groups - add, remove or sync (selected) groups.

Every Sync group contains all the buttons that modify it. The main toolbar's buttons modify the Sync View (the table that holds the Sync groups)

The rest of the buttons that modify inside the groups should reside on the group- level sub-menus (currently this sub-menu has a single button called "Sync this group" as well as three check boxes) . I would find it more intuitive should these sub-menus had the "plus" and "minus" buttons allowing additions or removals of the existing (selected) sync paths respectively.

See previous item.

On a different note, there is a potential risk inherent in all sync programs. I am referring to a scenario where one of the sync sources accidently gets deleted and the deletions are then propagated to all destinations resulting in a total data loss.

There is not such a risk. If a metadata files get deleted or corrupted or …, it will not get used at all.

I understand to this end you have the "archive" feature. However, since currently archiving is a manual process I wonder if you would consider turning the archiving on by default. Ideally, the archive should be viewed as the historic version of the current dataset existing for each sync group on the "server". That means that Kohighlights database would be included in every sync group by default as a server copy. In case of accidental data corruption one would be able to revert to the last good copy.

For me, the archived version are for the highlights that you want to have access to, without having to connect your reader. Something like locally saved to use whenever I need them. And with every sync it gets a copy of the most recent metadata.

In the future a cloud version of the program would make a good sense whereas the user would create a server copy as the very first step by synching his device to the server. Just an idea.

Hmm.. That will need a server (or more), and a knowledge of things I'm not currently interested in, and money (that I'm not making of this app). I think that, for an app that I made just to copy my highlights to the PC, it came a long way.. 😃

Thanks for your great work! I know this can be a lonely jorney. Let me know if you need any help with testing.

Thank you. I can send you a link to a beta version, when it's a little more on the ready side, to check (carefully)...

[edo-jan]

Thanks for clarifying. This is really helpful.

There is not such a risk. If a metadata files get deleted or corrupted or …, it will not get used at all.

Unfortunately, there is also a human error. It happened to me more than once when I accidently messed things up and deleted many annotations. Maybe it is because I am a professional pessimist I tend to view things through the prysm of recoverability or lack thereoff (I work in disaster recovery)... Lol

For me, the archived version are for the highlights that you want to have access to, without having to connect your reader. Something like locally saved to use whenever I need them. And with every sync it gets a copy of the most recent metadata.

I have not fully understood the last sentence above. Do you mean the program automatically updates the archive every time the group sync is run? Will there be a way to use the last good copy to recover should the said risk materializes?

Thank you. I can send you a link to a beta version, when it's a little more on the ready side, to check (carefully)...

Sure !

[noembryo]

Unfortunately, there is also a human error. It happened to me more than once when I accidently messed things up and deleted many annotations.

In KOHighlights? How did you do it? Manually? You can only delete individual annotations one by one. I couldn't think that this could happen by accident. There are always warning popups for the "human error".

And with every sync it gets a copy of the most recent metadata.

I have not fully understood the last sentence above. Do you mean the program automatically updates the archive every time the group sync is run? Will there be a way to use the last good copy to recover should the said risk materializes?

The metadata files that take part in the sync have different modified times. The annotations from the most recent is copied to the archive (of course only if you have the "Update to archive" checked). This copy is used as one part of the sync. Like the metadata files. That means that if you accidentally delete your annotations from your reader, syncing with the archive copy will recover them. Maybe I should name the checkbox "Sync with archive" or "Sync with archived"..

Edit: Changed it to "Sync with archived"

[edo-jan]

How did you do it? Manually?

Mostly manually. Don't underestimate the human propensity to screw things up and the speed with which the fingers can move ahead of the brain... lol. On a different occassion I lost nearly all comments (had to recreate most of them manually) after I moved away from kepub to epub on my kobo reader. I re-uploaded the new epubs from Calibre but could not restore the comments from the metadata files which was created for the kepubs...

That means that if you accidentally delete your annotations from your reader, syncing with the archive copy will recover them. Maybe I should name the checkbox "Sync with archive" or "Sync with archived".. Edit: Changed it to "Sync with archived"

Ok, that makes sense.

[noembryo]

Here is a beta version that has all the new features to test. Be careful with your files (always backup). The installer will remove v1.x.x installation files, but the settings will be preserved. Have fun.. 😉

[edo-jan]

Great! I have installed the beta and started testing. How do you want me to report the issues? Here is a couple so far:

1. The program doesn't see my android phone and I am not able to add it to sync (the phone is connected to windows via USB but the windows doesn't map it as a drive. Instead it shows in the explorer as "Card" and "Phone" for the SD card and the phone's internal storage respectively. (this was also an issue with the previous version of the program.)
2. I am unable to name / rename the groups. Right click / rename group does nothing. The dialog box doesn't provide a way to enter or edit the name

[noembryo]

Great! I have installed the beta and started testing. How do you want me to report the issues? Here is a couple so far:

Like you do now.. 😉 Just add them here and will fix it as we go..

1. The program doesn't see my android phone and I am not able to add it to sync (the phone is connected to windows via USB but the windows doesn't map it as a drive. Instead it shows in the explorer as "Card" and "Phone" for the SD card and the phone's internal storage respectively. (this was also an issue with the previous version of the program.)

This is a known issue, see #21

2. I am unable to name / rename the groups. Right click / rename group does nothing.

That's a nice catch.. 😃 If you select a path it will automatically get the title from the book, and then you can change it. The same applies if you create a group from a book in the Books view. But I forgot about a new entry.. 👎 Tomorrow will upload a new beta..

[noembryo]

Also don't leave a totally empty group while you're closing the app, or it will create a ghost group that won't be possible to change afterwards. Will be fixed in the next beta..

[noembryo]

Actually, I just uploaded a new beta with the fixes..

[edo-jan]

Just installed the new beta 1.9.9.1 The program did not save the old profiles (so I will need to recreate them manually). RE: not recognizing android devices. I am using a work-around: installed an FTP app on the phone and then mapped the connection as a local drive using NetDrive. Works well. Going forward for clarity I will refer to my WSA Koreader instance as "Device 1", My Kobo Libra2 reader as "Device 2" and my Android phone instance as "Device 3"). Or as D1, D2, D3 respectively.

Issues with 1.9.9.0 (will re-test this on 1.9.9.1 tomorrow):

Issue 1: New highlights do not show in the bookmark and comments are not synced

Starting point: D1 and D2 were in sync with each other, but not D3 Sync group created with the D1, D2, and D3. All the 3 check boxes checked (i.e. "sync positions", "merge highlights", and "sync with archived") Result: A dialog box informing that syncing of the group was completed. 1.1 D3 shows the new highlights from D1 / D2 but not the comments attached to the highlights.
1.2 D3 Koreader's bookmark menu doesn't show the new highlights or comments (only the old ones that were created on D3).

Issue 2: New highlights / comments are deleted from the originating device by subsequent sync and not propagated to other devices in the group.

Starting point: D1, D2 and D3 synced via Kohighlights 1.9.9.0 Sync group created with the D1, D2, and D3. All the 3 check boxes checked (i.e. "sync positions", "merge highlights", and "sync with archived") New highlights with comments made on D1, then Koreader on D1 is exited and the sync for the group is executed.

Result A dialog box informing that syncing of the group was completed. Re-opening the book on D1 shows that the newly made comment disappeared (tested twice, the same result) Opening the book on D2 and D3 doesn't show the new highlight / comments at all

[noembryo]

The program did not save the old profiles (so I will need to recreate them manually).

By profiles you mean Sync groups? If so, these are located in the settings folder of windows and are not erased. The same is true for the other settings of the app. BUT there was a bug that prevented the loading of all Sync groups if a Sync path was not existing. That is fixed in 1.9.9.2

• Point 1. A general advice regarding android/WSA I spend hours trying to understand why I was not syncing some files to Dropbox from my android mobile, only to found out that the changed file was changed only in the cache of the OS and was not committed to the disk. After that I had to explicitly check the files to be sure they were written before were send with any app. This is true for all Linux systems (random stack overflow comment) So, unfortunately you have to double check the existence of the changes in the .lua files..
• Point 2. Some general info about metadata formats. The current KOReader metadata format is going to be replaced (probably after the next update) with a "new" much better and cleaner format. I'll refer to the current format as the "old" format. This "old" metadata format had two different places to store highlights and bookmarks. It was build one new feature at the time, and this created a patchwork that was a little confusing and redundant. One issue that is currently affecting KOHighlights (that will be solved with the "new" format) is that the comments of the highlights are stored in the bookmarks place. So, when I find an existing highlight, the app continue without knowing if the comment is different. This is changed with the "new" format . Also note that "old format" bookmarks were not synced. Only highlights. (KOHighlights was also build with one feature at a time.. 😃 )

Issue 1: New highlights do not show in the bookmark and comments are not synced

If we rule our Point 1, then the culprit should be a bug that is fixed in1.9.9.2. Most of the new "merge" code was for the "new" format, but somehow, the old one was affected too.. 😞

Issue 2: New highlights / comments are deleted from the originating device by subsequent sync and not propagated to other devices in the group.

For the missing highlights, if we rule out the Point 1 from above, most probably happened because of the previous bug. For the missing/not updated comments, see Point 2.

This is the new 1.9.9.2.beta, with more squashed bugs.. Thank you for your time. 🙏

[noembryo]

Another day, another beta... version 1.9.9.3 😉

[edo-jan]

Point 1. A general advice regarding android/WSA I spend hours trying to understand why I was not syncing some files to Dropbox from my android mobile, only to found out that the changed file was changed only in the cache of the OS and was not committed to the disk. After that I had to explicitly check the files to be sure they were written before were send with any app. This is true for all Linux systems (random stack overflow comment) So, unfortunately you have to double check the existence of the changes in the .lua files..

I understand this has to do with how Koreader updates the metadata files. I was struggling with same issue before eventually learning that you can force the update by exiting Koreader from the exit menu. This causes Koreader to write the latest changes into the metadata.epub.lua (works on WSA and android phone)

Installed 1.9.9.3 Here are the issues so far: The program updated ok, but after I recreated the sync groups and executed the first sync group, the screen blinked, and the newly created groups disappeared.

Issues 1 and 2 remain unresolved.

Issue 3: No notification dialog is displayed upon execution of Merge/Sync

Starting point: D1, D2 and D3 synced via Kohighlights 1.9.9.0 Sync group created with the D1, D2, and D3. All the 3 check boxes checked (i.e. "sync positions", "merge highlights", and "sync with archived") New highlights with comments made on D1, then Koreader on D1 is exited to force metadata update and the sync for the group is executed.

Result No dialog box is displayed to notify about the sync results. Re-opening the book on D1 shows that the newly made comment disappeared Opening the book on D2 and D3 doesn't show the new highlight / comments at all

Issue 4: The "Books /Loaded" view fails to update the archive

Starting point: D1, D2 and D3 synced via Kohighlights 1.9.9.0 New highlights with comments made on D1, then Koreader on D1 is exited and the sync for the group is executed.

1. Go to the "Books /Loaded" view.
2. Use Scan Directory to navigate to the .sdr folder for a book (that is already in archive).
3. Select sync with archive
4. Go to the "Books/Archive" view and select the same book to see if the comment made on D1 did sync with the archive

Result: The comment is not synced with archive

5. Navigate back to "Books /Loaded" view Result: The book that was used to sync with the archive is no longer visible (disappeared from the view). I tried to use "scan directory" to redisplay it but it fails to see the book.

6. The comment made on D1 has disappeared (the Issue 2 persists)

Suggestion, consider changing the "Scan Directory" to "Select File" for consistency. In the Sync Group view one must select the specific .lua files, while in the Books/Loaded view the program scans the .sdr directory and selects the file automatically (or doesn't as per the issue above).

[noembryo]

Starting point: D1, D2 and D3 synced via Kohighlights 1.9.9.0

I'll try to recreate your test cases, but I have to warn you that this starting point is wrong. Version 1.9.9.0 had a bug that missed some highlights/bookmarks. These metadata are problematic. You have to start with the original metadata.

I'll post about the rest of your points a little later..

[edo-jan]

I'll try to recreate your test cases, but I have to warn you that this starting point is wrong. Version 1.9.9.0 had a bug that missed some highlights/bookmarks. These metadata are problematic. You have to start with the original metadata.

Agree. Let me reboot the machine and start from the beginning (I will restore the .sdrs from backup)

[noembryo]

There was a problem with the v1.9.9.3 too. 😳 Too many changes at the same time! Please, use this 1.9.9.4 beta instead..

[noembryo]

The program updated ok, but after I recreated the sync groups and executed the first sync group, the screen blinked, and the newly created groups disappeared.

This, I can't explain. Never seen anything like that. If you can replicate it, note down every step, so I can create it myself too.

Issues 1 and 2 remain unresolved.

These I think, have to do with corrupted metadata files. If it persists with 1.9.9.4 I'll check again. Same with Issue 3. The lack of info dialog at the end is fishy. Even if nothing was merged it should inform you about it. The Issue 4: What happened to the file? Was it deleted? If it was in the sdr folder, it should be scanned and discovered again. The only explanation I can think of is that some part of the app was crashed and it needed a restart to work properly again. Something is wrong here, but I have to wait for you to test it with 1.9.9.4 and normal metadata files, before starting a total rewrite.. I will probably need your metadata files too, if possible.

Suggestion, consider changing the "Scan Directory" to "Select File" for consistency. In the Sync Group view one must select the specific .lua files, while in the Books/Loaded view the program scans the .sdr directory and selects the file automatically (or doesn't as per the issue above).

The "Scan Directory" is a folder selector dialog that selects the root folder of your book library (or the root of your reader's exposed drive) to scan all the books' metadata files recursively. These can be hundreds or more. Selecting one file at the time (since usually there is only one metadata file in each .sdr) would make users spend a lot of time just to open a few books. The Sync groups on the other hand can have just a couple of files. Metadata files can have different names and not be alone in their own folder, depending on the device (some times many of them are located in a "History" folder), so choosing a folder is not enough to specify which file will get synced.

I'm anxiously waiting for your results..

[edo-jan]

Performed full uninstall of all older versions using Revo, Also removed all archive data, configs and cleaned up the registry. Removed all .sdr folders from the test books. Restored the .sdr from backup. Installed 1.9.9.4

Installation went ok. Was able to switch from / to new or old icons and themes. The new comments or highilights are not syncing. Both the Issue 1 and 2 persist. The newly made comments are not syncing to the destinations and are deleted from the origin

Here is another strange one: the same book on D1 is being recognized as a different book, so I am unable to sync this particular group. I have uploaded and re-apploaded it (using Calibre) trying to make it work, but no luck.

Issue 5: The metadata file of the same book is rejected as being from a different book

The three sdrs are attached. See if you are able to sync it. Book not recognized as same.zip

[noembryo]

Here is another strange one: the same book on D1 is being recognized as a different book, so I am unable to sync this particular group. I have uploaded and re-apploaded it (using Calibre) trying to make it work, but no luck.

Issue 5: The metadata file of the same book is rejected as being from a different book

The three sdrs are attached. See if you are able to sync it.

One possible reason for the books not recognized as the same, might be Calibre's reader. It used to be that if you opened an epub with Calibre's reader, it stored the reading position inside the epub. I don't know if this is still true, but if it is, the epub was changed and afterward, because of the different md5 checksums, KOHighlights identifies them as not the same books.

I'll check the metadata and come back to you. Thanks.

[noembryo]

More answers.. As I thought, the D1 book's md5 checksum is different from the others. That means that is somehow edited.

About the not syncing comments, the answer was in Point 2 of my previous comment. In a nutshell, comment editing can't be synced between the "old" (current) formats, but will be possible with the upcoming "new" metadata format.

What it will be most helpful are the metadata files that don't merge from the Issue 1 & 2, because whatever I do here, highlights are always merge as expected (except the comments of course)..

[edo-jan]

Maybe we should wait until Koreader releases the new version based on the new metadata format and then resume testing? Do the devs provide an ETA? I also hope they implement a framework that allows to sync comments irrespective of the screen resolution / size differences

D1 book's md5 checksum is different from the others. Not sure why this happens. It did happen before as well (with a different book in the old version). How can I make this file recongized? It is the same book, lol...

[noembryo]

Maybe we should wait until Koreader releases the new version based on the new metadata format and then resume testing?

I have to release KOHighlights before the KOReader release, so the users don't get confused. Anyway, the two formats have different code that must exist in the app for users of older KOReader versions. This has nothing to do with the Sync groups. These are a different feature all together.

Do the devs provide an ETA?

The release after the next one.

I also hope they implement a framework that allows to sync comments irrespective of the screen resolution / size differences

As I said the "new" format support it. And that's what they did. They re-wrote the metadata format structure. ~~Its me that have to implement the merging on KOHighlights. I just said that it's a lot of work to support it for the "old" format too. I might do it, but for now I'm working on this Sync group feature.~~ Edit: Done it..

So, can you send me the not merging metadata to see what's going on with that issues? I'm sorry for all the work I make you to do, but the beta testing is really appreciated.. 🙏

[noembryo]

A new 1.9.9.5.beta version is uploaded. I finally succeed to merge the different comments of the same highlight. The newer one will prevail and if the newest action was to delete the whole comment, it will get deleted from all. So time for a recapitulation.

• Issue 1. It should be resolved since the missing highlights were due to a 1.9.9.0 bug and the comments are synced now (from 1.9.9.5+)
• Issue 2. Same as Issue 1.
• Issue 3. This is something I can't reproduce. I need the specific steps.
• Issue 4. Books View merging works exactly as the Sync Groups merging. You didn't answer me about what happened to the lua file that was not scanned. I can't reproduce this.
• Issue 5. This is resolved, since the book file was edited, and it wasn't the same as the others.

Did I miss anything? If something from the above is wrong, please elaborate. Try the comment syncing to be sure..

Thanks 🙏

[edo-jan]

I will install and re-test it tonight. Thanks!

[edo-jan]

Performed full uninstall of 1.9.9.4. Also removed all archive data, configs and cleaned up the registry. Removed all .sdr folders from the test books. Restored the .sdr from backup. Installed 1.9.9.5 Recreated sync groups. I see that the book names for the sync groups are now populated automatically - this is great!

Issue 1 - appears to be fixed. New highlights and comments do propagate and displayed under the bookmarks

Issue 2 - appears to be fixed. The new highlights are no longer deleted from the originating device

However, there is a related problem which might be related (so I will insert it here as

Issue 2A - The deleted highlights are not propagated and restored back on subsequent sync

Starting point: Completed sync on a sync group made of D1, D2 and D3. All devices are in sync.

1. Open the book on D3 and delete one of the highlights.
2. Exit the Koreader to force sync
3. Return to KOHighlights and sync the group again.
4. Check the devices for the deleted highlight Result: The deleted highlight is back on D3 and is not removed from D2 and D1

Moving on:

Issue 3 - has never reoccurred since. I would close this as well

The Issue 4: What happened to the file? Was it deleted? If it was in the sdr folder, it should be scanned and discovered again. The only explanation I can think of is that some part of the app was crashed and it needed a restart to work properly again.

No the file was there, however the program could not see it. What you are saying above make sense. It is likely was caused by a crush of some part of the program. It can also be related to Issue 6 (see below). So I would close it for now.

Issue 4 - closed

Issue 5 - closed

I still have no explanation why this book changed md5 checksum... I had to copy the book from D3 to D1 and recreated the .sdr So it works now.

Issue 6 : After each sync the Books/Archived view fails to refresh showing duplicates of the same book

Starting point:

1. Clean install of the program v 1.9.9.5 (old version completely uninstalled, no settings preserved)
2. Create a sync group for a book on D1, D2, D3
3. Execute sync (with one checkbox checked)
4. Navigate to Books/Archived view

Result: The synced book's title is displayed.

1. Navigate back to the Sync Group views

2. Select another check box

3. Execute sync (with two checkboxes checked)

4. Navigate to Books/Archived view Result: The synced book is displayed twice

5. Navigate back to the Sync Group views

6. Select the third check box

7. Execute sync (with three checkboxes checked)

8. Navigate to Books/Archived view Result: The synced book is displayed three times

9. Now switch to the "Loaded view" and back to the "Archived" Result: the screen refreshes and correctly displays a single book

PS. There is no need to actually select one check box at a time. You can sync the same group several times - the result will be the same.

Issue 7 (suggestion, not a bug) By default check all options under "Show in Highlights" menu

Starting point - loaded or archived book selected (on a freshly installed program) The program's default setting for "Show in Highlights" are blank. Accordingly, the highlights pane only displays the chapter headings (see Issue 8)

Issue 8. Chapter headings cannot be removed from display even if everything is unchecked under "Show in Highlights"

Ideally, it would be great to define how many levels of headings one could display since this determines how the exported text looks in in .md format among other things.

[edo-jan]

Update on Issue 2A The same issue occurs (i.e. the deleted highlight is restored) when using a manual sync from the Loaded view with the archived. After the comment was deleted on D3 I exited Koreader and loaded the .sdr via the Books/Loaded view. It correctly showed that the highlight no longer was present. After syncing with the archived the deleted comment was restored...

[noembryo]

Issue 2A - The deleted highlights are not propagated and restored back on subsequent sync

Unfortunately this is something that can't be fixed. There is no way for the app to know if this missing highlight is because you purposely removed it or is not there yet and needs to get merged.

Update on Issue 2A The same issue occurs (i.e. the deleted highlight is restored) when using a manual sync from the Loaded view with the archived. After the comment was deleted on D3 I exited Koreader and loaded the .sdr via the Books/Loaded view. It correctly showed that the highlight no longer was present. After syncing with the archived the deleted comment was restored...

Are we talking about a deleted highlight or a deleted comment in a highlight? A deleted highlight will be merged from the other versions of the book. A deleted comment will get deleted from all other versions (if the deletion was made last)

I still have no explanation why this book changed md5 checksum...

Did you open it with Calibre?

Issue 6 : After each sync the Books/Archived view fails to refresh showing duplicates of the same book

I can reproduce this.. Really silly of my. Fix on it's way 😋

Issue 7 (suggestion, not a bug) By default check all options under "Show in Highlights" menu

Well, I already do this. All options should be checked.. Something is broken there.. I'll investigate. This must be connected with the Issue 8 too.

Very good findings, thank you.

[edo-jan]

Issue 2A - The deleted highlights are not propagated and restored back on subsequent sync

Unfortunately this is something that can't be fixed. There is no way for the app to know if this missing highlight is because you purposely removed it or is not there yet and needs > to get merged.

I wonder if this could be achieved via uni-directional sync. One day maybe? ...😏

Update on Issue 2A The same issue occurs (i.e. the deleted highlight is restored) when using a manual sync from the Loaded view with the archived. After the comment was deleted on D3 I exited Koreader and loaded the .sdr via the Books/Loaded view. It correctly showed that the highlight no longer was present. After syncing with the archived the deleted comment was restored...

Are we talking about a deleted highlight or a deleted comment in a highlight? A deleted highlight will be merged from the other versions of the book. A deleted comment will get deleted from all other versions (if the deletion was made last)

I have tested this with highlights only. My bad. I should not have used "highlights" interchangeably with "comments"

I still have no explanation why this book changed md5 checksum...

Did you open it with Calibre?

I used Calibre to copy the book from the Calibre library to Kobo Libra 2 as well as to Calibre Companion on WSA as well as on the phone

Very good findings, thank you.

I am glad to be of help. Thank you for all your work on KOhighlights. Keep up the good work!

[edo-jan]

I find the "Highlights" view somewhat difficult to use given that one cannot use it one book at a time (i.e. highlights from all books are presented in one big table). In the way of a suggestion it would be great if you could create a book selector pane similar to one used for the "Books" view? The book level metadata such as Title, Author, Type, Modified, etc. should stay in the book selector pane, and the Highlights table should contain the Highlihgts, Comments, Date, Page, Chapter (and possibly other levels of headings as per Issue 8)

[edo-jan]

Unfortunately the Issue 2 has returned after I have added the third sync group (I am not saying that this caused it since it might be just a coincidence). I have rebooted the machine and reopened the app. The sync groups survived but the issues persist. Interestingly enough, the new highlight (made on D2) shows correctly under the Books/Loaded view when I select its .sdr folder. Then it syncs correctly with the archived. Once the highlight or comment exists in the archived, the sync correctly propagates to other destinations (in this case D1 and D3). If it is not in the archived, it won't propagate and gets deleted from the originating device. 😟

[noembryo]

New 1.9.9.6.beta uploaded..

I wonder if this could be achieved via uni-directional sync. One day maybe? ...😏

😄

I used Calibre to copy the book from the Calibre library to Kobo Libra 2 as well as to Calibre Companion on WSA as well as on the phone

Well, something edited the book file for sure. You can see at the Book Info panel that there is an extra "-2 Testing" tag and a different language code.

Issue 6 : After each sync the Books/Archived view fails to refresh showing duplicates of the same book

Fixed in 1.9.9.6

Issue 7 (suggestion, not a bug) By default check all options under "Show in Highlights" menu Issue 8. Chapter headings cannot be removed from display even if everything is unchecked under "Show in Highlights"

There was a bug there too! Affected both. Fixed in 1.9.9.6

I find the "Highlights" view somewhat difficult to use given that one cannot use it one book at a time (i.e. highlights from all books are presented in one big table).

You can already do that if you use the Filter to limit the entries to that of a specific book.

Interestingly enough, the new highlight (made on D2) shows correctly under the Books/Loaded view when I select its .sdr folder. Then it syncs correctly with the archived. Once the highlight or comment exists in the archived, the sync correctly propagates to other destinations (in this case D1 and D3). If it is not in the archived, it won't propagate and gets deleted from the originating device. 😟

This I'm not sure I understand. You have an extra highlight in D2. If it is synced with the archived you see it in both loaded and archived. If you then merge it with the others, the highlight is copied to the others. If not synced with the archived and you merge D2 with the others, this highlight is deleted from all? Is this what you are saying? I can't replicate it by doing these steps here. Merging the books here always creates the new highlight in all of them, archived or not.. Am I missing something?

[edo-jan]

Interestingly enough, the new highlight (made on D2) shows correctly under the Books/Loaded view when I select its .sdr folder. Then it syncs correctly with the archived. Once the highlight or comment exists in the archived, the sync correctly propagates to other destinations (in this case D1 and D3). If it is not in the archived, it won't propagate and gets deleted from the originating device. 😟

This I'm not sure I understand. You have an extra highlight in D2. If it is synced with the archived you see it in both loaded and archived. If you then merge it with the others, the highlight is copied to the others. If not synced with the archived and you merge D2 with the others, this highlight is deleted from all? Is this what you are saying? I can't replicate it by doing these steps here. Merging the books here always creates the new highlight in all of them, archived or not.. Am I missing something?

This was in the context of the Issue 2 that has become an issue again. It occurred when syncing via the "sync group" however, when using Books/Loaded/Sync with Archived the highlight would sync with the "archived" correctly. If one uses "sync group" after that (that is when the "archived" contains the new highlight or comment) it would then correctly propagate to other devices in the group.

I will install 1.9.9.6 and test again. This time I wil let the program to update from the old version to make sure that the sync groups and other settings are preserved correctly.

[edo-jan]

Issue 9 - Installation of 1.9.9.6 triggered defender warning.

The program finished the install, but the sync groups are not preserved. Not sure if this was caused by the defender or not...

Issue 10 - The sync group settings are not preserved following the program update to new version

Manually restoring the "quarantined threat" and re-launching the program did not seem to change anything

[edo-jan]

I have re-created the sync groups manually. The Issue 2 is still causing an impact. Highlights / Comments are not propagated to other devices and are erased from the originating device

[noembryo]

Issue 9 - Installation of 1.9.9.6 triggered defender warning.

Unfortunately, nothing I can do here.. The way that I package all those setups is the same, using the same version of PyInstaller. From your screenshot I can see that the affected item is the KOHighlights Portable.exe, which is a packed .bat file that contains only the command ..\KOHighlights\KOHighlights.exe -p to start the app in portable mode. Its the exact same file that you had in every one of the betas. That means that Defender is loosing its marbles..

The sync group settings are not preserved following the program update to new version

Sync group setting are stored in the C:\Users\--USER_NAME--\AppData\Roaming\KoHighlights folder, in a json file called sync_groups.json. The only way that the app is deleting this file is if the user selects to "Remove any existing user settings" during the uninstallation of the app. During a normal install/update, nothing changes in this folder. You can keep track of this by yourself.

I have re-created the sync groups manually. The Issue 2 is still causing an impact. Highlights / Comments are not propagated to other devices and are erased from the originating device

Since I can't reproduce this, is it possible for you to send me these 3 metadata files (with one having this extra highlight) to check it here?

[edo-jan]

The only way that the app is deleting this file is if the user selects to "Remove any existing user settings" during the uninstallation of the app. During a normal install/update, nothing changes in this folder. You can keep track of this by yourself.

Not sure where I can select "Remove any existing user settings" or not to remove... here is the screen shot.

And here is what I see when I click "Next"

[noembryo]

Not sure where I can select "Remove any existing user settings" or not to remove..

"Remove any existing user settings" is an option that you see if you Uninstall the app. During installation you only have the option to remove the "Program files" directory..

[edo-jan]

"Remove any existing user settings" is an option that you see if you Uninstall the app. During installation you only have the option to remove the "Program files" directory..

I see what you mean. However, most users are not used to this type of "orderly" process when one is expected to uninstall the app first and only then install the new version. People are conditioned to run the new version on top of the old one and expect that the program will do the rest - i.e. unistall the old version preserving the settings and install the update. Unless the program follows the expected behavior I am afraid many users would find themselves in the same situation as I did - i.e. needing to recreate the sync groups, etc.

[noembryo]

People are conditioned to run the new version on top of the old one and expect that the program will do the rest - i.e. unistall the old version preserving the settings and install the update.

And this is exactly what the setup is doing. Nothing more. I use the same installer in many programs. Not once were the user settings affected by the installation of a newer version. I just did an new install, created a Sync group, closed the app. Run the setup again, open the app, the group is there. Again, more groups, more installations, the settings persist.

I don't understand what is happening there. Windows are going bonkers!! 🙄

[edo-jan]

I have re-created the sync groups manually. The Issue 2 is still causing an impact. Highlights / Comments are not propagated to other devices and are erased from the originating device

Since I can't reproduce this, is it possible for you to send me these 3 metadata files (with one having this extra highlight) to check it here?

For clarity, this only impacts the highlights. The comments seem to work ok. The attached are the 3 .sdrs. The D2 has a couple of highlights not present in D1 and D3. The sync erases the highlight from D2 instead of propagating them to D1 and D3. This happens to other books as well so you should be able to replicate Apr 24 - highlights deleted from origin not synced.zip

[edo-jan]

I don't understand what is happening there. Windows are going bonkers!! 🙄

I agree, windows is getting worse and worse.

I just did an new install, created a Sync group, closed the app. Run the setup again, open the app, the group is there. Again, more groups, more installations, the settings persist.

I wonder if this was caused by the Defender that kicked in during my previous installation... Let me try again

[noembryo]

Apr 24 - highlights deleted from origin not synced.zip

These files have exactly the same highlights You should send me the ones that aren't synced..

[edo-jan]

These files have exactly the same highlights You should send me the ones that aren't synced..

I don't know what is going on. Before sending it to you I made sure that D2 was different (I did not open the .lua directly, but instead used the Books/Loaded view to scan the .sdr. It then showed the two extra highlights. I then sent the files to you. Now when I looked at the zipped .sdrs sent to you - they are exactly the same... 😟😟😟

[edo-jan]

Upon further reflection. It appears that this can happen because of either: a) Koreader did not update the metadata file, so even the highlight was present on the screen, but not yet written in metadata.epub.lua b) KOHighlights is somehow using a catched verison of metadata and not refreshes the file prior to sync

Regarding (a): Koreader has a setting that impacts how frequently metatada file is refreshed. However, my understanding is that it always updates it before the program is exited. This is what I have been using all this time to force metadata updates.

Regarding (b): I don't how KOhighlights does it - so maybe you could see if there is a potential issue there... Any other ideas?

[noembryo]

Regarding (b): I don't how KOhighlights does it - so maybe you could see if there is a potential issue there... Any other ideas?

Every change that the sync/merge does to the metadata is in-place (in the memory). So the metadata are always up to date. I manually update the table's column values, so that they are also up to date, without the need of a new scanning of the files.

Of course the scanning of the file while is loaded is the most reliable..