Relative references for home directory for syncing between computers

[kd2flz]

I attempted syncing my foliate data between 2 computers. I thought this would work, because the downloaded epubs are all stored in foliates "books" directory. However, when I try to open the file from Foliate's library, I get the error "could not open file; file not found." The progress information, cover, and other metadata is transferred sucessfully - in fact, when I navigate to where the file is stored and select "open with foliate", the book opens in the correct place (where I left off reading on my other computer). The only problem I can see is how Foliate references the book - for instance How I Built This is referenced as file:///home/[user]/.var/app/com.github.johnfactotum.Foliate/data/com.github.johnfactotum.Foliate/books/urn%253Auuid%253A9a5fd66a-97f8-4e7b-baf0-215c4a9cd12b@2021-01-21T01%253A28%253A26%252B00%253A00@Guy%2520Raz_How%2520I%2520Built%2520This.epub It turns out the problem I have is that I am using a different username on the second computer.

Describe the solution you'd like Ideally, the book location would sync between multiple computers with different usernames. This works in other apps (one I use this syncing method with is Cozy). Basically, I use syncthing to sync the entire directory between my devices. If Foliate used a relative reference for the home directory (ie: /home/$USER or similar), then this problem could likely be avoided.

Describe alternatives you've considered An obvious solution is just to "reimport" all my books on the second device. Once the device knows where the file is stored, it should be able to sync the position without a hitch. However, this is a bit clumsy, as it would mean importing each book at least twice into my library.

Additional context If there is a better way to sync my library between devices, feel free to let me know! This is just the first way that came to mind.

[johnfactotum]

Yes, currently the library does not give any special treatment for files in $HOME or even Foliate's own books directory. Actually when the library was built, the URI references (com.github.johnfactotum.Foliate/library/uri-store.json) was never intended to be synced at all, and was expected to be different on each device and thus excluded from syncing. The URI store is also only used if the book is not indexed by Tracker. See https://github.com/johnfactotum/foliate/issues/119#issuecomment-517890810.

I guess it would make sense to reference files in $HOME differently. This should be easy to implement and there aren't really any downsides.

[elsiehupp]

I had been meaning to make an issue similar to this a while back. Basically, importing books into Foliate doesn't seem to give Foliate "ownership" of the files, which would a major convenience for not having to deal with file management.

If you manually copy a book into Foliate's library folder and then import it, Foliate still doesn't treat the book as something it owns, so if, for example, you rename the user folder (which, yes, is a real thing I have done), it breaks all of the local file references within Foliate.

Generally the workflow I would most prefer is when importing books (e.g. ePub files) having the option of Foliate copying the books into its internal store and managing them from there so that I don't have to maintain a copy of the books elsewhere.

FWIW the backing store doesn't even need to be the filesystem. It could just as well be a SQL database or something else entirely, which might provide some performance benefits, though the implementation would be somewhat more involved.

Eventually it would be nice if there were some way to sync a Foliate library with an ereader or smartphone, but that's kind of a whole 'nother ballgame considering the lack of standardization for a lot of library metadata.

[johnfactotum]

for example, you rename the user folder (which, yes, is a real thing I have done), it breaks all of the local file references within Foliate.

It doesn't break if Tracker is enabled in Foliate (which it is by default) and all your files are indexed by Tracker.

Tracker indexes files and automatically tracks whether they are moved or deleted. When opening a book from the Library view, Foliate will ask Tracker for the path of the file (by using its unique identifier); if the file is not indexed by Tracker, for whatever reason, Foliate will then use the URI of file from the last time you opened the book.

Generally the workflow I would most prefer is when importing books (e.g. ePub files) having the option of Foliate copying the books into its internal store and managing them from there so that I don't have to maintain a copy of the books elsewhere.

I suppose copying and syncing à la Calibre is simpler to manage, but it's too heavy handed for people (myself included) who use Foliate more as a viewer than a book manager. But going forward I'm not really sure what would be the best approach. See also my comments to #730.

[elsiehupp]

This happened when I was setting up a friend’s computer with Elementary Linux after his Windows bricked itself. The Foliate instance was installed via Flatpak.

When I changed the username (because he didn’t like the one I initially picked for him) what I did was create a second account with the correct username, create and log into a temporary third account, move the contents of the first account’s home folder to the second account’s home folder, recursively chmod the new contents of the second account’s home folder, then delete the first account.

This process was presumably well beyond the usual use case, and along with Foliate being a Flatpak install it may have contributed to breaking the references despite Tracker. Basically Foliate was still looking for the ePub files in /home/moe/.var/... rather than in /home/ali/.var/..., though the library itself (i.e. the database containing the broken references) loaded correctly when I opened Foliate.

FWIW Tracker is kind of a complete mystery to me. A while back I was trying to get it to work as a Spotlight backend over Samba, but the documentation was basically nonexistent, and the Samba folks strongly encourage the use of Elasticsearch instead. When I was using Ubuntu, Spotlight over Samba “just worked”, but on Debian for whatever reason it doesn’t. Obviously this isn’t particularly relevant here; I’m just kvetching about the unreliability of filesystems as persistent application data stores.

Regarding the approach of copying the ePubs into an application-controlled directory, part of the issue is probably the decision by the Flatpak developers to use an obfuscated directory location rather than putting application directories somewhat front and center the way iCloud Drive does. macOS is janky in its own special ways, but the sleight of hand they use to allow developers to make application-controlled directories user-accessible is relatively well implemented.

I suggested a potential fix over at gvfs and Flatpak, but it hasn’t seemed to be a priority for anyone.

[johnfactotum]

Ah, yes. Flatpak. While it's possible to use Tracker 2 with Flatpak, Foliate currently doesn't support it. Tracker 3 supports Flatpak sandboxing better and we'll probably add support for it once we port to Tracker 3.

Regarding the approach of copying the ePubs into an application-controlled directory, part of the issue is probably the decision by the Flatpak developers to use an obfuscated directory location rather than putting application directories somewhat front and center the way iCloud Drive does. macOS is janky in its own special ways, but the sleight of hand they use to allow developers to make application-controlled directories user-accessible is relatively well implemented.

This has nothing to do with Flatpak. Without Flatpak, the data will live in hidden directories like ~/.local/share or ~/.config.

To "make application-controlled directories user-accessible" sounds like a contradiction to me. "Application-controlled" implies "accessible only through the app" (If not, what does "control" mean?), which implies that it's not "user-accessible (outside the app)".

In any case, Foliate's primary feature is viewing e-books, which is why it tries its best not to "own" those files if possible. In the future I'll probably turn the Library closer to what GNOME Books is today—primarily a frontend for viewing the list of files from the system's Tracker miner. Tracker has been and still is a core part of GNOME and is their intended way for apps to implement library-like features, so there's really not much point in not using it.

To be honest, though, a part of me still wants to return Foliate to be more in line with its original goal: a simple viewer, like Evince. Actually, I originally considered naming it "Convince", as a sort of companion app for Evince, and to maybe convince GNOME developers to add EPUB support in Evince...

[elsiehupp]

An example of an application-controlled directory that’s visible in the user’s home folder is Nextcloud, which uses a hidden database stored in each subdirectory (alongside inotify) to keep track of changes.

The way application-specific iCloud folders work is that they’re essentially a scratch disk for the application sandbox that facilitates bringing data in and out via the Finder. Applications that use iCloud usually store most of their data in obfuscated directories and databases, while the iCloud folders are used for very specific things like documents and browser downloads.

Basically what I’m suggesting is that Foliate (and other Flatpak applications) could have the option of making one part of their sandbox more easily accessible within Nautilus. In Foliate’s case, this would be a folder where ePubs with pretty filenames would be stored. Foliate would keep track of external changes within that folder and indicate if a “recently opened” ePub was no longer available, as well as automatically noticing any new ePubs added to the folder.

tl;dr the way to think of iCloud Drive application folders is that they’re like the Flatpak Portal equivalent of a thumb drive or, if you’re old, a floppy disk. Does that make sense?

…

As an aside, I feel like it would make sense for there to be a standard user-level directory for ebooks in, e.g., user-dirs.dirs, just like there is for, say, music. IIRC Android has a standard user-level directory for podcasts, so it isn’t that strange to imagine expanding the list judiciously. There are several open issues suggesting additional directories on the Freedesktop Gitlab, but they don’t seem to be getting acted on.

Additionally, it’s not entirely clear to me how these XDG directories interface with Flatpak, as opposed to the more generalized XDG_CONFIG and the like.

…

FWIW if the Foliate “library” view just showed recently opened ePubs and ePubs added to the one folder it watches, that could still make sense in terms of not being classically a “library” with a full database backend.

It would still be nice if Foliate continued to provide an OPDS frontend, if only because there aren’t a lot of other options, and it would make sense for the OPDS frontend to share the same “library” folder, which might otherwise be inconveniently sandboxed.

Similarly, I don’t remember how much ability Foliate has to edit embedded ePub metadata, but within reason that seems like a useful feature to have, even if it’s not front and center. (Compare, for example, the basic, classic iTunes metadata popup, though it would also be nice for this to be functionality to be available in some Nautilus extension.)

For editing book content rather than metadata, though, it does seem like it would make more sense to facilitate opening the Calibre book editor rather than implementing the functionality directly.

One thing that can be maddening with Apple’s design philosophy is that they’ll implement fairly deep functionality but heavily obfuscate it from the main user interface. By contrast, a lot of GNOME applications simply don’t have the functionality that they don’t put front and center, probably because GNOME doesn’t have the dragon’s hoard of funding that Apple does.

(An example in this case could be quietly allowing users to convert MOBI and AZW3 ebooks to ePub while not heavily advertising it. Like, basically, Foliate could automatically convert MOBI and AZW3 books that it opens, then save the new ePub book into its own library folder, in much the same way that I was suggesting copying external ePub books into the library folder, or at least offering to do so, when they are opened in Foliate. The conversion could be entirely non-customizable, with a directive that users wanting more flexibility can download Calibre to use alongside Foliate. It’s not unusual for applications to offer import and sometimes export of formats they don’t natively support, though AZW3 in particular is very, very similar to ePub in structure.)

…

Anyway, apologies for rambling, lol.

[kd2flz]

While I respect your project vision to build an eBook viewer, I have to say, you have built a damn good eBook manager/reader in the process! I’m not aware of anything this good in the Linux world, and certainly not in Windows, where your choices are essentially Kindle for PC or Calibre. While Calibre is a great project, it is more functional as an eBook syncing solution than a reader. I guess what I’m trying to say is that if it seems like people are trying to use the project for something you didn’t really intend (managing eBooks), it’s because the project is already genuinely good at doing this. Honestly, this is how I use the app—I have multiple computers which I use for reading eBooks, and I like being able to sync my progress between then. The interface is fantastic, and I imagine that the use case of cross-device syncing will only grow with the growth of devices like the pine phone, and now the Pine64’s announcement of an eInk device.

Anyways, I mostly just wanted to share my appreciation for this genuinely great app and try to shed some light on what is probably a pretty common use case.

From: John Factotum @.> Sent: Wednesday, August 18, 2021 2:25 PM To: johnfactotum/foliate @.> Cc: David Rhoads @.>; Author @.> Subject: Re: [johnfactotum/foliate] Relative references for home directory for syncing between computers (#739)

Ah, yes. Flatpak. While it's possible to use Tracker 2 with Flatpak, Foliate currently doesn't support it. Tracker 3 supports Flatpak sandboxing better and we'll probably add support for it once we port to Tracker 3.

Regarding the approach of copying the ePubs into an application-controlled directory, part of the issue is probably the decision by the Flatpak developers to use an obfuscated directory location rather than putting application directories somewhat front and center the way iCloud Drive does. macOS is janky in its own special ways, but the sleight of hand they use to allow developers to make application-controlled directories user-accessible is relatively well implemented.

This has nothing to do with Flatpak. Without Flatpak, the data will live in hidden directories like ~/.local/share or ~/.config.

To "make application-controlled directories user-accessible" sounds like a contradiction to me. "Application-controlled" implies "accessible only through the app" (If not, what does "control" mean?), which implies that it's not "user-accessible (outside the app)".

In any case, Foliate's primary feature is viewing e-books, which is why it tries its best not to "own" those files if possible. In the future I'll probably turn the Library closer to what GNOME Books is today—primarily a frontend for viewing the list of files from the system's Tracker miner. Tracker has been and still is a core part of GNOME and is their intended way for apps to implement library-like features, so there's really not much point in not using it.

To be honest, though, a part of me still wants to return Foliate to be more in line with its original goal: a simple viewer, like Evince. Actually, I originally considered naming it "Convince", as a sort of companion app for Evince, and to maybe convince GNOME developers to add EPUB support in Evince...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/johnfactotum/foliate/issues/739#issuecomment-901333367, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALGVCJGLH5HN6HVMB7CRF3LT5P3G7ANCNFSM46GDBSQQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://mail1.ccistack.com/fmlurlsvc/?fewReq=:B:JVI2PTk5Nip6MT4iPCplaDE8PTY8PSp/ZWtibXh5fmkxaD87PDtpaDluPTg0Oz5qOm08OzU1bT04Pj4+Oj45PTU0bjVqPjU9OCp4MT06PjU/PT08NDQqfWVoMT07RUVDYGNtPD04Ojw+IT07RUVDYGNvPD04Ojw+Kn5vfHgxaG16ZWh+ZGNtaH9MYW1lYH94bW9nIm9jYSpvMT01KmRoYDE8&url=https%3a%2f%2fplay.google.com%2fstore%2fapps%2fdetails%3fid%3dcom.github.android%26utm_campaign%3dnotification-email.

[johnfactotum]

Foliate would keep track of external changes within that folder and indicate if a “recently opened” ePub was no longer available, as well as automatically noticing any new ePubs added to the folder.

That's exactly the function of Tracker. It allows apps to track changes to files.

If a file is user-accessible, it means that the user is supposed to use a file manager (or any other app) to do stuff with the file. So that means the app doesn't actually own or control the file, as any other apps can do anything it wants to the file at any time.

This means that the app needs to track those files whether they are put in a folder that the app "owns" or not. So there isn't actually any real advantage in having an "EPUB" folder; a file indexer would track files in exactly the same way whether they are in one folder or another, or even scattered across many folders.

In simpler cases, apps can track files by simply querying the list of files on the filesystem. This is what a file manager does, for example. But apps often require metadata beyond the basic info (such as filename and modified date) provided by the filesystem. This means that in order to track files efficiently, they need to index those files in advance, with something like Tracker.

If an app can track files without a Tracker-like process, that means it tracks files in the same way a simple file manager does. But that would mean that it offers no advantage over a file manager. Therefore, any app that provides a library-like feature requires Tracker or equivalent technology. Significantly, many file managers also employ file indexers. For example, Nautilus provides full text search with Tracker. This means there's often little point in duplicating the work of a file manager. Here, though, we may observe the advantage of a system indexer like Tracker over each app having its own indexer: the data can be shared among different apps, saving resources.

Honestly, this is how I use the app—I have multiple computers which I use for reading eBooks, and I like being able to sync my progress between then. The interface is fantastic, and I imagine that the use case of cross-device syncing will only grow with the growth of devices like the pine phone, and now the Pine64’s announcement of an eInk device.

Progress syncing does not require library-like features to work. Foliate didn't have a Library view until v2.2.0, but syncing has been working since the initial release.

In fact, prior to having a Library view, syncing actually worked better, in the sense that this particular issue ("Relative references for home directory for syncing between computers") did not exist back then — Foliate did not store any path references at all.

Indeed, I'd like to think that syncing was made possible precisely because Foliate did not have a library view. If it did, it might store all it's data in an opaque database instead of plain flat files, and might use paths instead of identifiers or hashes. These would make syncing more difficult.

An example where this mechanism can be powerful (or at least flexible): notice that the files are referenced by its URI rather than simply a path. This is because Foliate supports opening remote URLs directly. So you don't even have to sync the e-books themselves. You can sync reading progress and annotations while keeping the e-books on a cloud or NAS.

[kd2flz]

Interesting, thanks for all the background! I might try saving the books in a remote drive.

---

From: John Factotum @.***> Sent: Thursday, August 19, 2021 2:26 AM To: johnfactotum/foliate Cc: David Rhoads; Author Subject: Re: [johnfactotum/foliate] Relative references for home directory for syncing between computers (#739)

Foliate would keep track of external changes within that folder and indicate if a “recently opened” ePub was no longer available, as well as automatically noticing any new ePubs added to the folder.

That's exactly the function of Tracker. It allows apps to track changes to files.

If a file is user-accessible, it means that the user is supposed to use a file manager (or any other app) to do stuff with the file. So that means the app doesn't actually own or control the file, as any other apps can do anything it wants to the file at any time.

This means that the app needs to track those files whether they are put in a folder that the app "owns" or not. So there isn't actually any real advantage in having an "EPUB" folder; a file indexer would track files in exactly the same way whether they are in one folder or another, or even scattered across many folders.

In simpler cases, apps can track files by simply querying the list of files on the filesystem. This is what a file manager does, for example. But apps often require metadata beyond the basic info (such as filename and modified date) provided by the filesystem. This means that in order to track files efficiently, they need to index those files in advance, with something like Tracker.

If an app can track files without a Tracker-like process, that means it tracks files in the same way a simple file manager does. But that would mean that it offers no advantage over a file manager. Therefore, any app that provides a library-like feature requires Tracker or equivalent technology. Significantly, many file managers also employ file indexers. For example, Nautilus provides full text search with Tracker. This means there's often little point in duplicating the work of a file manager. Here, though, we may observe the advantage of a system indexer like Tracker over each app having its own indexer: the data can be shared among different apps, saving resources.

Honestly, this is how I use the app—I have multiple computers which I use for reading eBooks, and I like being able to sync my progress between then. The interface is fantastic, and I imagine that the use case of cross-device syncing will only grow with the growth of devices like the pine phone, and now the Pine64’s announcement of an eInk device.

Progress syncing does not require library-like features to work. Foliate didn't have a Library view until v2.2.0, but syncing has been working since the initial release.

In fact, prior to having a Library view, syncing actually worked better, in the sense that this particular issue ("Relative references for home directory for syncing between computers") did not exist back then — Foliate did not store any path references at all.

Indeed, I'd like to think that syncing was made possible precisely because Foliate did not have a library view. If it did, it might store all it's data in an opaque database instead of plain flat files, and might use paths instead of identifiers or hashes. These would make syncing more difficult.

An example where this mechanism can be powerful (or at least flexible): notice that the files are referenced by its URI rather than simply a path. This is because Foliate supports opening remote URLs directly. So you don't even have to sync the e-books themselves. You can sync reading progress and annotations while keeping the e-books on a cloud or NAS.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/johnfactotum/foliate/issues/739#issuecomment-901645043, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALGVCJELURDC3YBJRB7N233T5SPYPANCNFSM46GDBSQQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://mail1.ccistack.com/fmlurlsvc/?fewReq=:B:JVMxOj4+MS19NjklOy1ibzY7OjE7Oi14Ymxlan9+eW42Om0+bjs6Mj5pODhvPzI+OTxuMmo/aWk8Mz8+aDg8MzhpMjs+PT0/bS1/Njo9OTI4Pj84PD4temJvNjo8QT1aTT1hOzs6PTo8Jjo8QT1aTT1nOzs6PTo8LXloe382b2p9Ym95Y2Rqb3hLZmpiZ3h/amhgJWhkZi1oNjoyLWNvZzY7&url=https%3a%2f%2fplay.google.com%2fstore%2fapps%2fdetails%3fid%3dcom.github.android%26utm_campaign%3dnotification-email.

[elsiehupp]

This means that the app needs to track those files whether they are put in a folder that the app "owns" or not. So there isn't actually any real advantage in having an "EPUB" folder; a file indexer would track files in exactly the same way whether they are in one folder or another, or even scattered across many folders.

iCloud app-specific folders exist for two very specific reasons:

1. They’re a part of the application’s sandbox, and on iOS the sandbox has zero direct access to the outside filesystem (unlike on macOS, where you can grant it on an app-by-app basis).
2. If you’re using iCloud, the folder facilitates keeping data synchronized between the macOS, iOS, and iPadOS versions of a given application. (In practice, most applications use third-party backends because iCloud sync doesn’t support, e.g., syncing with corresponding Android or Windows clients.)

I guess in the context of Flatpak, a user-accessible application-specific folder serves the following purposes:

1. As with Apple, it’s part of the app sandbox. (And, lacking a more obvious XDG place to store eBooks, it provides an immediate option.)
2. Assuming one or another of the long-running Linux mobile projects ever becomes substantially usable, the Flatpak sandbox could also facilitate system-level per-application cloud sync (which doesn’t really exist right now outside Apple’s and Google’s respective walled gardens).

Outside Flatpak, a user-accessible application-specific folder doesn’t really make sense except as way of working around the limitations of the xdg-user-dirs specification (i.e. the lack of a standardized $XDG_EBOOKS_DIR variable).

[johnfactotum]

As with Apple, it’s part of the app sandbox.

The sandboxing problem is mostly solved in Tracker 3. The app doesn't need any filesystem permissions. It talks to Tracker through a portal and everything just works. Tracker 3 also has some data separation. For example, you can give an app permission to see document files through Tracker but not video or music files.

In practice, though, people are concerned that Tracker might not be installed on the system. So they use their own copy of Tracker inside the sandbox, instead of talking to the system instance of Tracker. This means that it will require permissions to specific folders. (This is essentially how Tracker 2 works with Flatpak.) This would essentially be an "app-specific directory".

It is, however, usually considered confusing and inconvenient to have a hardcoded directory, and is, at least in my observation, probably the biggest complaint people have about Tracker based apps, as more traditional library apps typically allows you to add any folder to the library, and many people keep their files in various locations, including external drives.

Assuming one or another of the long-running Linux mobile projects ever becomes substantially usable, the Flatpak sandbox could also facilitate system-level per-application cloud sync

How does it facilitate syncing?

Outside Flatpak, a user-accessible application-specific folder doesn’t really make sense except as way of working around the limitations of the xdg-user-dirs specification (i.e. the lack of a standardized $XDG_EBOOKS_DIR variable).

The user can create a folder called "Books" in Home if they want. What would be the advantage of standardizing? The currently standardized directories are already of dubious practical utility. Many people do not actually place, e.g., their music files in ~/Music, and if required to do so, must resort to symlinks.

But symlinking is also a very simple solution if one really wants to make app specific data directories accessible to the user: one can, for example, make ~/Books a symlink for ~/.var/app/myapp/data/mybooks. This is probably much easier than messing with gvfs. I imagine it would also be very easy for file managers to detect such symlinks and make a special case for their icons. After all, all app icon names are the same as the app's ID, so it would be trivial to generate a special folder icon based on the symlink target. The main work is probably adding an option in Flatpak manifests to allow making such a symlink upon installing the app.

[elsiehupp]

Giving each application a single directory to keep track of simplifies syncing by being aggressively dumb. While something like Tracker is of course far technically superior, you have identified the main issue, namely that it isn’t consistently available. The main advantage with dumb standardization is that it creates a more consistent development target that is more likely to “just work”. (Of course, when things don’t “just work” on an Apple OS, hoo boy, that’s a huge pain in the ass. Where Linux will give you a firehose of information, Apple will get you smoke signals, if that.)