Enhancement request - sync jobs and other features

[edo-jan]

The following is inteface / feature ideas you may consider for future development. The proposed interface is inspired by an old nifty program called "allway sync". The main feature allows creating sync jobs for each book. Each job resides in its own tab where it can be configured and run.

The second file show the sync direction configurator. This is how it is implemented in allway sync. Extremely flexible since one can select the sync direction, participating devices, as well as weather or not to propogate changes, deletions, or both.

The "Analyze" button produces the summary of changes for each device and the "Synchoronize" button - executes the job.

I hope this is in line with the direction you are taking your great little program. Thanks again for your service to the community!

The following shows some of the sync configuration options that one can change upon clicking on the "change" button:

[edo-jan]

Every change that the sync/merge does to the metadata is in-place (in the memory). So the metadata are always up to date. I manually update the table's column values, so that they are also up to date, without the need of a new scanning of the files.

I can be wrong but re-scanning the files before every sync/merge is albeit a tiny bit slower, but would be a lot more reliable, especially when the changes are made while Kohighlights is running in the background.

[noembryo]

This new 1.9.9.7.beta might be the last one, since I don't think we have any standing issues (apart from the Windows quirks).

[edo-jan]

Installed the latest beta 1.9.9.7 . For the first time the program displayed a dialog box asking if I wanted to remove the settings. This is certainly a positive. The initial loading of the program took a long time. The sync groups were present, however, some of the entries were in red (even though all devices were connected). Since there is no "refresh" button the only way for the program to recognize the connected devices was to remove them from the sync groups and add them back. When for the first time I switched to the Books/Archived view the program displayed a dialog informing that it is loading the database and the program appeared to be frozen for a long time (certainly more than a minute, may be close to a minute and half) when it finally displayed the archived view. But here is the odd thing (see the pic):

For whatever reason the archived view doesn't see two out of three books. I made highlights for those books but they failed to sync. And the archived view still show a warning saying that "epub is missing" I noticed that the two failed ones have a diffrent path which I don't know how to interpret this since the path of the archived view is supposed to point to the database, not a book location on a drive.
The last but not least: when I went back to the Sync Groups view this is what I see 😟😟😟

[edo-jan]

Question: do you really need the program to be backwardly compatible with the [old] metadata framework that Koreader is replacing? In my view trying to accommodate both standards in a single program introduced an extra layer of complexity. From my limited understanding of the domain (please pardon me for being overly simplistic here) it would appear that it maybe best to keep the old working version of Kohighlights as is, and develop the new functionality as a separate branch tailoring it to the new metadata format in the upcoming version of Koreader. You can call it differently as well - e.g. "KoHighSync" or something like that to differentiate from the old one... Just a thought.

[noembryo]

Installed the latest beta 1.9.9.7 . For the first time the program displayed a dialog box asking if I wanted to remove the settings. This is certainly a positive.

This dialog was always displayed when you uninstall the app. What is new, is that now there is another dialog that asks you if you want to uninstall the app. This will appear only when updating from 1.x.x.x versions to 2.x.x.x. The beta is still 1.9.9.x, so you will see this with every beta install.

The initial loading of the program took a long time. The sync groups were present, however, some of the entries were in red (even though all devices were connected).

Positioning the mouse over the red text will provide a tooltip that describes the specific error. If the error is a missing path, then the specific path was missing at the time the groups were loaded. But it might be a different error.

Since there is no "refresh" button the only way for the program to recognize the connected devices was to remove them from the sync groups and add them back.

You could close/re-open the app for the groups to get reloaded. No need to recreate them..

When for the first time I switched to the Books/Archived view the program displayed a dialog informing that it is loading the database and the program appeared to be frozen for a long time (certainly more than a minute, may be close to a minute and half) when it finally displayed the archived view.

This should be almost instantaneous for that many entries. Something else was blocking the disk.

But here is the odd thing (see the pic):

For whatever reason the archived view doesn't see two out of three books.

You can always check the Path column to see if the specific path exist in explorer. If it says its missing, well, it must be missing.. 😉

I made highlights for those books but they failed to sync.

I don't understand what you're saying here. I need some more info..

And the archived view still show a warning saying that "epub is missing" I noticed that the two failed ones have a diffrent path which I don't know how to interpret this since the path of the archived view is supposed to point to the database, not a book location on a drive.

The data of the book is in the database. The path is the path to the book that is copied to the database. That means that when you archived this book, the actual book was located in an path that now does not exist.

The last but not least: when I went back to the Sync Groups view this is what I see 😟😟😟

That means that there was something wrong with the data of the specific Sync group. If you send me the sync_groups.json that is located in the C:\Users\--YOUR_USER_NAME--\AppData\Roaming\KoHighlights directory, I can tell you exactly what the error is..

[noembryo]

Just uploaded this 1.9.9.8.beta version that adds a refresh button on every Sync group. I had it there from the beginning, but I didn't thought that anybody would need it.. 😏

[edo-jan]

I have tested 1.9.9.8 beta. Unfortunately, Issue 2 is not yet resolved. 😟

adds a refresh button on every Sync group. I had it there from the beginning, but I didn't thought that anybody would need it.. 😏

This is useful - thanks!

[noembryo]

I have tested 1.9.9.8 beta. Unfortunately, Issue 2 is not yet resolved. 😟

Issue 2: New highlights / comments are deleted from the originating device by subsequent sync and not propagated to other devices in the group.

As we already discussed, highlights that are deleted from a book will get re-created after merge, because there is no way of knowing if the highlight was deleted or not yet merged. OTOH, comments that are deleted from one device should get deleted from all the devices. If this is not happening I'll need some files that reproduce this, so I can investigate..

[edo-jan]

As we already discussed, highlights that are deleted from a book will get re-created after merge, because there is no way of knowing if the highlight was deleted or not yet merged.

I did not make my self clear. I was refering to comments and highlights that were (1) made on the originating device and (2) deleted from that device after sync. Also there is no propagation to other devices in the sync group. That is the behavior first described under Issue 2.

[noembryo]

If you are referring to this..

I have re-created the sync groups manually. The Issue 2 is still causing an impact. Highlights / Comments are not propagated to other devices and are erased from the originating device

Since I can't reproduce this, is it possible for you to send me these 3 metadata files (with one having this extra highlight) to check it here?

I still don't have any files that reproduce this. The ones that you send me was already synced. Without files that exhibit this problem, there is nothing I can do, since my trials here do not behave this way..

[edo-jan]

I still don't have any files that reproduce this. The ones that you send me was already synced. Without files that exhibit this problem, there is nothing I can do, since my trials here do not behave this way..

The problem is seem to be occurring during the run time. It is hard for me to explain so I am going to record the screen session to illustrate. Here is the first recording - unfortunately the sync groups disappeared again so I will recreate them shortly and continue. In the meantime here is what I have: For clarity, I did try closing / openning the program, and even reinstalling it. The groups are gone.

https://github.com/noembryo/KoHighlights/assets/165089603/13736ea4-cc3e-4ebe-9233-72ee2a847c6b

[noembryo]

Can you please send me the sync_groups.json that is located in the C:\Users\--YOUR_USER_NAME--\AppData\Roaming\KoHighlights directory, and are not showing up?

[edo-jan]

I have already recreated the groups so the .json might be reflecting this. But, it is attached sync_groups.json Here is the video illustrating the Issue 2

https://github.com/noembryo/KoHighlights/assets/165089603/6680119e-4507-4bab-b269-1b4ab2273376

[edo-jan]

Here is a better illustration

https://github.com/noembryo/KoHighlights/assets/165089603/0ece0eaf-ddf2-4c7a-852c-8a4b81e36bc3

[noembryo]

OK. One picture is 1000 words. So imagine what a video is.. 😃 Now I know what happens, so I will describe the steps of the events so you can understand it too.

• KOHighlights opens.
• The Sync Groups are loaded and read the metadata files from the disk to memory.
• KOReader is opened.
• A new highlight is made.
• KOReader closes and the metadata file is saved on the disk.

At this point the highlight is written in the metadata file (that file I wanted).

• Switching to KOHighlights that doesn't know about the new addition.
• Start the syncing of the group.
• KOHighlights uses the already loaded data to synchronize the files and saves them to disk.

At this point the files are over-written and are not containing the additional highlight.

• Re-opening KOReader shows no highlight as expected.

The solution to this might be the refresh button (I'll have to check it though). Did you press it? If, after the creation of the new highlight you were pressing "Refresh" the groups should be reloaded and the data would be updated. Any syncing after this should work as expected. I have to test this more.

But the most reliable solution of course would be to close and re-open KOHighlights.

And an of topic question: What is the ftp app you are using to show the Android directories to the PC?

[noembryo]

Just checked Refresh and it works as it should. If you press it after any external change, the data of the files are updated to memory and the sync keeps the new highlights..

[edo-jan]

OK. One picture is 1000 words. So imagine what a video is.. 😃

It sure is better than words... 🤩

Switching to KOHighlights that doesn't know about the new addition.

yep, I think this is it!

The solution to this might be the refresh button (I'll have to check it though). Did you press it? If, after the creation of the new highlight you were pressing "Refresh" the groups should be reloaded and the data would be updated. Any syncing after this should work as expected. I have to test this more.

I have not used the refresh. I should do some more testing as well.

ust checked Refresh and it works as it should. If you press it after any external change, the data of the files are updated to memory and the sync keeps the new highlights..

Great. Although you should consider incorporating the refresh into the sync routine

And an of topic question: What is the ftp app you are using to show the Android directories to the PC?

On Android I am using File Manager as an FTP server https://play.google.com/store/apps/details?id=com.alphainventor.filemanager&hl=en_CA&gl=US&pli=1

And on windows NetDrive to map the FTP connection as a drive letter https://www.netdrive.net/

[noembryo]

Great. Although you should consider incorporating the refresh into the sync routine

That's a good idea. I added it. 👍 Is there anything else that is not working at this time?

[edo-jan]

After I manually refreshed the group the sync for that specific group did not display a notification about completion. The comment got deleted anyway. Subsequent closing and reopening of the program resulted in disappearance of the sync groups again. The attached .json is FYI. Something is not right. sync_groups.json

[noembryo]

The sync_groups.json opens normally here. With missing path of course, but it opens. There is something different there. Can you try to disable the Defender checks while doing the tests?

[edo-jan]

I will try to do some more testing tomorrow

[noembryo]

OK. Tomorrow I'll try to upload a newer beta.. 🙏

[noembryo]

Also, can you please send me an archive with any error_log_2024-04-xx.txt you'll find in the C:\Users\--YOUR_USER_NAME--\AppData\Roaming\KoHighlights directory? After every group sync there should be a popup to inform you about a success or a failure. The absence of one shows a program crash. These error text hacks might show us what happened (although I haven't implement a normal logging into the app yet).

[edo-jan]

Also, can you please send me an archive with any error_log_2024-04-xx.txt you'll find in the C:\Users--YOUR_USER_NAME--\AppData\Roaming\KoHighlights directory?

Here we go error logs KOHighlights.zip

[noembryo]

Well, now we're getting somewhere..

I can see some errors there, but since I have uploaded many different betas, it is difficult to find all of them in the code. Did find one that can be fixed, and there are some older ones that they shouldn't happen now, but the main discovery is a lot of PermissionError: [Errno 13] Permission denied errors happening while the app tries to read some metadata file. This inability to read the data might create all the consecutive errors. That means that someone is locking some metadata file and KOHighlights can't read it. That someone can be WSA, Defender, Windows or some random other app. WSA is the only one that I don't have here, so that might explain a lot..

This is the 1.9.9.9.beta that has the "auto reload groups before sync" feature that we discuss earlier. You delete the current error_logs (when KOHighlights is closed of course), so we can start from scratch and see what errors will occur. If anything strange happens on your side, you send me the new error_logs we can either fix the errors or at least recognize who is the culprit..

For a real-time check on what is happening, you can open the About KOHighlights dialog and switch to the log tab..

[noembryo]

There is an irrelevant error in this version, that confuses the book extensions with the metadata ones. Don't pay attention to that..

[noembryo]

Because I found another error, I have uploaded a new 1.9.9.10.beta that fixes both of them...

[noembryo]

.. and another one.. 1.9.9.11.beta with lots of changes under the hood.

[edo-jan]

Do you want me to test this with the defender on or off?

[noembryo]

You can try it with the Defender on as you normally work. If any problem happen, then we have to check different configurations. And read the error_logs..

[edo-jan]

1.9.9.11 installed ok. Prior to the installation I deleted the old error logs. The program displayed the sync groups and archived books. Proceeding to test: Opened Koreader on D1 (Kohighlights is running in the background), Made a comment on a book Exited Koreader Back to Kohighlights Sync / Groups view. First attempt to sync the group did not display the "completed" dialog box . Results: The archived file doesn't show the new comment. Re-opened the book on D1 and the comment is still there (at least this time it is not deleted) Here is the error log error_log_2024-04-27.txt

[edo-jan]

Closed and re-openned Kohighlights. One of the sync groups has disappeared from the Sync /Groups view (the one that was tested above is still there). Here is the .json sync_groups.json

[noembryo]

Can you send me the error_log after the not loading of the groups?

Edit to add: It seems that one (or both) of the Calibre_Companion metadata could not be loaded. Try to copy the .sdr folder to another location and sync them from there. This should work.

[edo-jan]

Can you send me the error_log after the not loading of the groups?

here we go. I also tried syncing again - but the result was the same (no dialog) error_log_2024-04-27.txt

[edo-jan]

Try to copy the .sdr folder to another location and sync them from there. This should work.

Strange thing hapenned. I copied both android .sdr files to C:/ drive location and created another sync group for the same book (see the screenshot). Then I executed the sync on the second (new) group. It displayed the completion dialog box! After that I tried syncing the old group (then one that used to fail) and ... it also synced... 🤯🤯🤯

[noembryo]

It's the same error that happens when syncing or loading the sync groups. It reads the first metadata (from "F:\Macarius the Great\FIFTY ...") and when it tries to read the next ("G:\Calibre_Companion\FIFTY...") it gets nothing. Did you try to copy the .sdr folder someplace else (after the edits) to use these as sync paths? It's most probably the android paths that are getting locked...

[edo-jan]

Did you try to copy the .sdr folder someplace else (after the edits) to use these as sync paths?

Yes, I copied the .sdrs to C - see the post above (the screenshot shows that the second group has paths pointing to C drive). The strange thing is that after syncing the newly created group it also synced the .sdrs from the android devices

[edo-jan]

it reads the first metadata (from "F:\Macarius the Great\FIFTY ...") and when it tries to read the next

Actually, I have not tried syncing the book you mentioned. I synced "F:\Isaac the Syrian..." not "F:\Macarius the Great\FIFTY ..." it seems that you are looking at the wrong place in the log. The Macarius book was the one that disappeared from the Sync / Groups

[noembryo]

The paths are from the sync_groups.json second group and I used them as an example. Here is yet another 1.9.9.12.beta that I re-enable the "Permission denied" errors that I had previously disabled. Using this might tell as what file is getting locked..

[noembryo]

BTW, the groups are always there, even if the paths are locked. If you close the WSA sessions they will get loaded (with red, missing paths of course)

[edo-jan]

BTW, the groups are always there, even if the paths are locked. If you close the WSA sessions they will get loaded (with red, missing paths of course)

Let me try this and see if the missing book re-appears. I will then do a full uninstall with Revo removing all traces of 1.9.9.11 and earlier. Then I will do a clean install of 1.9.9.12

[noembryo]

I will then do a full uninstall with Revo removing all traces of 1.9.9.11 and earlier. Then I will do a clean install of 1.9.9.12

This is not necessary, since the setup installs only the Program files folder. Nothing special is written to the registry (except the uninstall info/paths etc.) And the settings that I told were there are..

[edo-jan]

I shut down WSA and reopened Kohighlights. Both "Isaac the Syrian" groups are there but "Macarius" book is not there. I opened the .json file to see if it is there - it is not there either. So it has disappeared for good

[noembryo]

The "Macarious" group was in the sync_groups.json you upload a few messages back. Because there weren't loaded when you created the new one (because of the locks), they didn't get saved again.

[edo-jan]

installed 1.9.9.12 Recreated the Macarius group.
Opened the book, made a comment. Exited Koreader Went back to Kohighlights Tried to sync the group same issue (no dialog informing about completion). The error log says there was a crash (see attached) error_log_2024-04-27.txt sync_groups.json

[noembryo]

Unfortunately no extra info other than before. The same missing metadata. Copying the .sdr folders most probably will work as before..

[noembryo]

A new 1.9.9.13.beta that's inform you if a file can't be read..

[edo-jan]

A new 1.9.9.13.beta that's inform you if a file can't be read..

The windows explorer can read all the files ok. The "file lock" you mention seems only to occur during the run time. I am clearly not an expert on this and I don't know what the refresh routine does, but I wonder if it can be further enhanced by forcing the program to perform some operation on the file such as reading its attributes, etc. .. In other words to force the android release the lock if it is there (just as it "releases" it when the file is selected in windows explorer). I speak as an amateur so pardon me if this doesn't make sense.

[noembryo]

Unfortunately, what I'm doing is just accessing the files to read their contents. The problem is that the files should be in a really peculiar state. A simple non accessible file will produce a simple error and I could just inform you that I can't access it. What this is doing is that sometimes it lets me access the file and then locks the process and doesn't let me read it. I don't understand how this is possible, but it's enough low level for me to move it beyond the scope of this simple app. Perhaps I could do more if there was an easy way to reproduce that, but at the moment there isn't.

[edo-jan]

version 1.9.9.13 displayed warning about inability to read one of the files. This allowed me to use the select button to re-add the file to the group (at this point it displayed a dialog saying that the file was already in this group). Following this it did sync ok! So we are getting somewhere. Unfortunately, after the second group synced successfully the program lost the groups again. I tried closing and reopening the program , but alas... Since the groups are stored in .json located on C:/ drive this is something not related to android locks. Once the sync group disappearance issue is addressed (or you find a way to reload the .json from the interface) we should probably be content with the build and resume testing once Koreader releases the new metadata version. Do you have any ideas why the groups keep disappearing?

error_log_2024-04-28.txt sync_groups.json