maintenance: rclone as shared library

x0b commented 3 years ago

TLDR: I intend to move RCX from the current process based model when controlling rclone towards an in-process shared library model.

When RCX was initially conceived as rcloneExplorer, the only way to control rclone was by using the command line interface. Since then, rclone has grown in features, among them an remote control (RC) interface to invoke most rclone actions. Since this is a purpose-build interface for inter-process communication, the semantics and data formats are much better defined. Using the RC interface also avoids the overhead of having to spawn a new rclone process for each an every action.

The new interface is already built and is used e.g. in #8 (VCP). However, it currently comes with the significant disadvantage of using a TCP/IP transport, which introduces overhead and new failure modes. It also requires workarounds around the Android app lifecycle model, which you see by the requirement of the background service notification in v1.12.

The next step will be to load rclone directly as a shared library. Experiments towards this goal have been underway for the last two years, but recent changes in rclone, go and our tooling have made that goal much more achievable than before. A minimal demo app built with this mode is already running.

There are a few big tasks still remaining until we reach this goal:

[ ] and Implement and stabilize a native library interface in rclone (see comments in #123)
[ ] The RC interface is missing a way to perform OAuth (related: https://github.com/rclone/rclone/issues/3455)
[ ] The RC interface is missing a way to enable rclone serve (issue: https://github.com/rclone/rclone/issues/4505)
[ ] Add rc interface for rclone cat and rclone rcat over named pipes and/or unix domain sockets.

There is also lots of work to do on the RCX side - while the RC interface is mostly ready, we need to migrate all the old parts of the app to using the new parts of the app. We also need to find ways to deal with the resulting changes towards the rclone execution and lifecycle.

Other related issues:

69: Keep SFTP connections open with rclone rc (if possible)

ncw commented 3 years ago

Note that the work integrating rclone as a library has this issue https://github.com/rclone/rclone/issues/4891

x0b commented 3 years ago

Yes, the first prototype this year was based on https://github.com/rclone/rclone/issues/4891#issuecomment-786587698 modified for gomobile.

x0b / rcx

maintenance: rclone as shared library #130

69: Keep SFTP connections open with rclone rc (if possible)