Progress Sync with KOReader

[captn3m0]

Is your feature request related to a problem? Please describe. I would like to sync my progress between KOReader and Foliate

Describe the solution you'd like

Foliate would sync the progress with the KOReader Sync Server. The sync server is minimal, supports multiple users and can be self-hosted.

As I read the book, the progress would be pushed the Sync Server. Any other application (KOReader, or another Foliate instance) can download the sync progress and automatically update that when they open the book.

Describe alternatives you've considered Writing a script that would read the Foliate database, and sync the progress with the koreader-sync-server instead.

Additional context

I think the KOReader sync server is decent enough to be standardized and used across other applications. It is FOSS, and can be easily self-hosted. Having a common system that works across multiple such applications will be a great plus to the ecosystem.

[johnfactotum]

I think this should probably be a plugin (if we ever get a plugin system... probably not in the short term) or a standalone script. Adding a sync server is a bit too much trouble, and would actually be more restrictive than the current behavior.

Currently you can easily sync between "Foliate instances" with any file syncing solution you like (see wiki for more info). You can even use Git to version control your progress, or even resolve progress and annotations conflicts on different devices with Git (at least in theory; in practice since the JSON files are currently saved without line breaks, this is perhaps not the best way to do it).

Syncing from Foliate to KOReader should be straightforward, since they seem to be using EPUB CFI (not byte offsets like many other readers). You simply have to calculate the MD5 hash for the file yourself before syncing, as Foliate does not calculate/store such a hash. To sync from KOReader, you'd need to get the unique identifier of the file.

[captn3m0]

Thanks. I was planning to write a script for now, but thought I'd open an issue here regardless.

To sync from KOReader, you'd need to get the unique identifier of the file.

Could this be done when I open the file? Since foliate will know the hash then. But I need a way to hook to foliate file reads then.

[johnfactotum]

Could this be done when I open the file? Since foliate will know the hash then. But I need a way to hook to foliate file reads then.

There's no way to do that currently, unless you modify Foliate's source, which is why a plugin system would be really helpful.

For now as a workaround you can try opening the file with Foliate (you can run com.github.johnfactotum.Foliate --add [FILE] to "open" the file headlessly), and then find the most recently modified file in ~/.local/share/com.github.johnfactotum.Foliate. The name of this file will be the URI-encoded identifier.

[antrmn]

As of today, there's no way to use the standardized EPUB CFI mechanism to sync reading progress across different reading apps. This is really a pity.

[johnfactotum]

The problem here is that there isn't a standardized way to uniquely identify a book.

In theory, you'd think that you'd be able to use the unique identifier for this purpose (which is what Foliate does), but in practice this could be problematic, and the spec actually recommends against this, and it offers no standardized alternative.

Even among reading systems that use hashes, they don't use the same algorithm. And when they do, they don't necessarily calculate them in the same way. E.g. they might only hash part of the file up to a certain size.

[johnfactotum]

Actually I forgot about library/uri-store.json. It's a map from the unique identifier to the URI of the file. Also, you can query Tracker with the identifier if your files are tracked with Tracker.

So basically, you would

1. Watch .json files in ~/.local/share/com.github.johnfactotum.Foliate. Any change would mean that the book was opened by Foliate.
2. The name of the JSON file is the URI-component-encoded unique identifier.
3. Use library/uri-store.json or Tracker to find out where the file is actually stored.
4. Compute the hash from the file.
5. To sync to Foliate, store a list of hash-identifier correspondence yourself.

This way, it's possible to build an external program that could do progress sync with any reader ~that uses CFIs.~ Since the external program will have access to the file itself, it can do its own conversion between different fragment selectors/locators. So you're not even limited to CFIs.

[johnfactotum]

And I was wrong: KOReader doesn't use CFIs. It uses something called xpointers, which is basically just XPath. Converting between these two probably isn't too hard, but it does require loading and parsing the EPUB.

[ConEstArthur]

I want this same functionality. However it's a pity that syncing highlights isn't possible.