Add initial rclone support

[ScottESanDiego]

Here's a first swag at adding rclone support.

Good:

• Kinda sorta works!
• I think it supports subdirectories even (didn't fully test this, but seemed to in my quick test)

Bad:

• Includes an rclone binary (should really wget it from the static URL on the rclone site)
• Hard-codes the path to the rclone config file as kobo_rclone.conf in the kobocloud directory
• Abuses the kobocloudrc syntax since you cannot really identify an rclone path via URL

To make this work, you need to do a couple things:

1. Install rclone somewhere, and create a new config for it (e.g., "ScottE_DropBox:eBooks")
2. Copy the .rclone.conf to the kobo, and put it in .add/kobocloud/kobo_rclone.conf
3. Add a line to kobocloudrc that looks like RCLONE Foo (e.g., RCLONE ScottE_DropBox:eBooks)

Depending on how many books you have, the first sync can take a LONG time. Afterwards, rclone is pretty good about only transferring new files, so it goes quickly.

Full disclosure, I didn't yet 'make' a new KoboRoot.tgz with this, I was just updating files in the existing tarball for my tests.

[ScottESanDiego]

I should have listed another "Bad" as it puts some config bits into the getRcloneFiles.sh versus config_kobo.sh like it should. Easy to fix, I'm just throwing this together.

[fsantini]

I suggest bundling a binary of rclone with the package and then using rclone itself to auto-update from the website.

[ScottESanDiego]

So we'd ship a rclone.conf that had a config pointing to https://downloads.rclone.org/rclone-current-linux-arm.zip and have the install step use that canned config to download the new version? Interesting idea!

I think rototilling to enable all that is a bit more than I have cycles for at the moment. (I'm still trying to debug why newly sync'ed books aren't shown, and require a reboot for the Clara HD to import them - regardless of how they were downloaded. I suspect the loopback mount isn't doing what we think it should.)

[fsantini]

As far as I understood, rclone can still be used as a normal command-line tool (like curl or rsync). So we don't need to change the configuration file format, we would just need to adapt the get.sh (or getRemoteFile) to use rclone,

[ScottESanDiego]

Hmm, I'm not sure I've ever used it that way. I've only used it via rclone config and then specify a target after that, based on what was named in the config. Do you have a pointer to how to use it one-shot (ala curl or whatever)?

[fsantini]

https://rclone.org/commands/rclone/ For example I believe that rclone copy would do what we need.

[ScottESanDiego]

I'm not so sure. rclone copy source:sourcepath dest:destpath assumes source:sourcepath isn't a URL, it's either a local path or it's a reference to the rclone.conf name (e.g., ScottE_DropBox:eBooks).

Trying to pass a URL doesn't work, as it's looking in the config file:

~> rclone copy https://github.com/fsantini/KoboCloud/pull/35 .
2020/07/07 16:27:39 Failed to create file system for "https://github.com/fsantini/KoboCloud/pull/35": didn't find section in config file

If you create a config file with rclone config, you end up with a conf file that contains something like:

[ScottE_DropBox]
type = dropbox
token = {"access_token":"BLAH","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}
client_id = BLAHBLAH
client_secret = BLAHBLAHBLAH

Then when you pass rclone copy a path, it looks at source as the key in that config file, and sourcepath as a directory at that destination.

Anyway, we can probably make something work. Worst case, use curl and unzip to grab https://downloads.rclone.org/rclone-current-linux-arm.zip locally, or worst-worst case, keep bundling the rclone binary like we do with curl now (which is what I did in this PR).

[fsantini]

You are right. Rclone does not support download from anonymous links. Mmm this is quite a bummer because it would indeed change the whole approach.

[ScottESanDiego]

I still think we can make it work though. For downloading rclone itself, we can install a fixed rclone_update.conf file and ship a version of rclone (or something like that).

For downloading books, having the user create their own rclone.conf (as I describe in the PR) is trivial, and they can put it next to kobocloudrc when they are editing that.

I really think the solution as-implemented in this PR works really well, it just needs some cleanup (aka, what I list in "Bad" in the PR) to be perfect. It certainly is WAY easier and faster (after the initial sync) than the current "iterate through every URL and download with curl" method.

[ScottESanDiego]

Clarified process + another user w/success using the above in https://github.com/fsantini/KoboCloud/issues/36

[AndreaTosti]

It took me a while to figure out how to use it but in the end it was worth it. Tested on Kobo Libra H2o, it also supports sub-folders. Thank you @ScottESanDiego Very important: you must not put more than one RCLONE Foo correspondence in kobocloudrc because the last of these always applies.

I seem to have understood that to delete a book, you must do it directly on the cloud storage.

[ScottESanDiego]

Oh, that's interesting. I would have thought kobocloudrc would use all of the items not just the last one. Hrm, that's an interesting find! I assume unrelated to the rclone part though?

RE deleting files, what works for me is removing them from the cloud storage, and the next time rclone runs it removes them from the Kobo.

[AndreaTosti]

Oh, that's interesting. I would have thought kobocloudrc would use all of the items not just the last one. Hrm, that's an interesting find! I assume unrelated to the rclone part though?

Assuming kobocloudrc's content is

RCLONE dropbox:Folder0
RCLONE dropbox:Folder1

I tried to run the following commands locally:

rclone sync --config ../kobo_rclone.conf --no-check-certificate dropbox:Folder0 outputDirectory
rclone sync --config ../kobo_rclone.conf --no-check-certificate dropbox:Folder1 outputDirectory

The first rclone populates the outputDirectory folder with the contents of Folder0. The second rclone deletes the contents of the outputDirectory folder, populates the outputDirectory folder with the contents of Folder1 Eventually you end up with only the content of Folder1.

In my opinion the "flaw" is related to the use of rclone, while this problem does not occur with curl

I could also be wrong, of course.

[ScottESanDiego]

Ah, I see it now. Good catch.

[AndreaTosti]

Ah, I see it now. Good catch.

To get around this "problem" it would be enough to do rclone copy instead of rclone sync.

[ScottESanDiego]

But then deleting wouldn't work - you'd have to delete items locally on the Kobo I think.

I think what we'd need is a way to specify the output directory for the rclone sync in the kobocloudrc.

[AndreaTosti]

But then deleting wouldn't work - you'd have to delete items locally on the Kobo I think.

I think what we'd need is a way to specify the output directory for the rclone sync in the kobocloudrc.

EDIT2: you are right about deleting files, because with rsync copy you should delete, in order, first the files from the cloud, then in the ebook

Thinking about it on the fly, since the format of the arguments is always remote: path, it would be enough to extract the path into $path variable and use ${outDir}/${path} as the destination folder.

EDIT: remains the case where path is an empty string in which I do not know in which folder to put the files

EDIT3: if the path is empty, then the contents of the whole main folder will be deleted and populated with new content (synced). Just let the user know. If the path is not empty, then only the directory with the same name will be replaced (synced).

[sobkas]

And it works: https://github.com/sobkas/KoboCloud/commit/6808c636fabee307e7313b2f0b76277d6bcb971d Didn't add my local_test.sh because without rclone config file it's useless

[krillin666]

Is there any update on this ? This is essential for anyone reading academic PDF using Zotero. This would let us use ZotFile to send the pdf to kobo, annotate on KoReader and get the modified pdf file back into Zotero !

[marklar423]

@ScottESanDiego I think this is a wonderful solution that would open the door to the many sources rclone supports, without having to code for each independently. And it would add two way sync, as others mentioned.

Regarding the need for a user to make a .rclone.conf file: what if the user put their rclone config inside of kobocloudrc, with either a prefix for each line or beginning + ending markers?

This way get.sh can simply write out a .rclone.conf file and we'd at least keep all config local to a single file, which might be easier for people.

[ScottESanDiego]

That seems like a good solution @marklar423 . I've kinda dropped this project, since DropBox via NickelMenu has been working "good enough" for me now, and @fsantini seemed to want to go a different path with the KoboCloud bits.