Open tcf909 opened 4 years ago
Initial investigation #2882
Ideally, if we could push events to the kernel that would simplify everything, but:
Ultimately, the kernel events are consumed by tools like inotifywait which are just outputting a delimited stream of events. rclone wait could easily be a drop in replacement (and a standard) for outputting a delimited stream across a diverse set of backends.
The only backends supporting ChangeNotify today are
For this proposal to be useful we need more backends to support it. Some backends (eg dropbox) have the cabability but it isnt supported. Some backends (eg most of the others) don't have the capability.
So for this to be worth putting the effort in, we need a ChangeNotify fallback for backends which don't support it.
I think the most important backend to support would be the local backend - https://github.com/rclone/rclone/issues/4152
I think your point 2 an rclone wait
command is a good idea. Point 3 has come up before.
Point 1 would be quite tricky.
I'm not particularly happy with the ChangeNotify internal interface
https://github.com/rclone/rclone/blob/61e4b4db429a175e9e4437c93127929aa9373269/fs/fs.go#L559-L562
It is hard to use and the events it generates aren't well defined so I think this could do with a rethink problably!
Proposal: rclone wait [fs:/] (+monitor subsystem)
Problem Statement:
Currently there is no clear API / approach to watch for changes and act on them when using rclone. Filesystems each have an event notification system, but given that rclone transparently works with remote backends as well as local backends one cannot simply use inotify (on linux) for example.
rclone faces a unique challenge in that it abstracts the complexities of working with many types of backends (including many different types of local filesystems).
To make matters worse, the linux kernel / libfuse (and I'm assuming other kernels) does not offer any way (without patching the kernel) to inject events into the kernel for a FUSE file system. This negates the ability for rclone to work with the existing ecosystem of tools and approaches as a simple producer of events (while letting existing tools that typically rely on the kernel to consume events).
Proposed Solution:
1) Consolidate change events internally from backends to allow rclone functions to monitor for events while rclone is running (mount, copy, sync, move, etc..). In general present the internal code with a standard API to monitor for events at a directory level (recursive) or file level across any supported backend (remotes that support delta change updates, local filesystems that support notification events).
Current State:
Feedback:
Looking for community feedback to stage for a future implementor.