The existing cloud backup function layered rsync on top of rclone. The issue with this approach is that it:
Required the use of a fusermount via rclone and then synchronization was handled by rsync. rclone has built-in sync functionality for remote destinations, so the use of rsync on top of it introduced unnecessary complexity.
The existing method of using rclone to temporarily mount a remote to /storage/cloud relied on caching for the virtual file system. This led to errors because the script would copy the files and rsync would begin synchronizing them, but the files hadn't always finished transferring from the temporary cache to the cloud mount. Additionally, the use of caching could eventually lead to the accumulation of stale data in the cache folder.
Approach
One option would have been to simply disable caching for the VFS so files had to be written directly to /storage/cloud before rsync would begin. However, after testing, this proved to significantly slower sync times. So, I decided to create a new backup experience that didn't depend on rsync but still left everything related to rclonectl in place. This is because I wanted to ensure that the dev team felt good about this approach, and it means that changes can be rolled out incrementally. For example, I've changed how cloud backups work, but have not modified cloud restore or the ability to save to the cloud on exit.
Results
I've gone through and tested this on my RG353M and everything is working well:
An error message is displayed if rclone hasn't been configured yet
The new script automatically will select the first entry in rclone.conf if multiple remotes have been set up
It is verified to work identically as before with the same file type inclusions and exclusions.
It is incredibly speedy, to the point where I added in time so users could read the success message before the script closes. This 2 second delay could be removed to make the process even faster.
I've only been able to test this on my RK3566 device, but I've updated the dependent packages and it should function the same on any aarch64-based build. However, additional testing would be great.
r3claimer
commented
1 week ago
Background
The existing cloud backup function layered rsync on top of rclone. The issue with this approach is that it:
/storage/cloudrelied on caching for the virtual file system. This led to errors because the script would copy the files and rsync would begin synchronizing them, but the files hadn't always finished transferring from the temporary cache to the cloud mount. Additionally, the use of caching could eventually lead to the accumulation of stale data in the cache folder.Approach
One option would have been to simply disable caching for the VFS so files had to be written directly to
/storage/cloudbefore rsync would begin. However, after testing, this proved to significantly slower sync times. So, I decided to create a new backup experience that didn't depend on rsync but still left everything related torclonectlin place. This is because I wanted to ensure that the dev team felt good about this approach, and it means that changes can be rolled out incrementally. For example, I've changed how cloud backups work, but have not modified cloud restore or the ability to save to the cloud on exit.Results
I've gone through and tested this on my RG353M and everything is working well:
rclone.confif multiple remotes have been set upI've only been able to test this on my RK3566 device, but I've updated the dependent packages and it should function the same on any aarch64-based build. However, additional testing would be great.