search

hwittenborn / celeste

GUI file synchronization client that can sync with any cloud provider
GNU General Public License v3.0
1.23k stars 41 forks source link

Only sync when files are changed #34

Closed hwittenborn closed 1 year ago

hwittenborn commented 1 year ago

Currently we sync by just looping forever and checking for changed. It would probably be better to just detect when a directory has been changed and then sync that directory.

geometry1 commented 1 year ago

I tested Flatpak Celeste with Nextcloud. This results in one CPU core constantly at 100%, which apparently is precisely due to the problem described in this issue of continuous syncing. I hope this can be solved soon.

Great project by the way, very promising! I am looking forward to get rid of the standard Nextcloud Sync Client that causes me a number of problems.

hwittenborn commented 1 year ago

I've looked into various Rust libraries and I haven't found anything that works well. It looks like we might be able to just call rclone lsl on the directory being synced (it seems to be pretty fast in my testing, even on large directories), and then just compare timestamps for everything inside of that to determine what needs synced.

loveisfoss commented 1 year ago

I've looked into various Rust libraries and I haven't found anything that works well. It looks like we might be able to just call rclone lsl on the directory being synced (it seems to be pretty fast in my testing, even on large directories), and then just compare timestamps for everything inside of that to determine what needs synced.

It sounds good to me, at least until a better solution is suggested. A preliminary step like "Calculating differences..." and then the actual sync process.

gabrielelucci commented 1 year ago

What about inotify?

hwittenborn commented 1 year ago

I've tried that a bit in the past @gabrielelucci, but it causes issues when any file in the specified directory can't be read.

I.e. if the directory ~/Documents is being synced, any file that's unreadable by the current user will cause a panic to be raised. There's a few ways to work around that part at least, but if a directory is unreadable and then later becomes readable, the inotify crate won't pick up those changes.

If I use use something like rclone lsl, I can just record the timestamps of every file that gets shown, and then the next time it checks timestamps it can just compare the new timestamps to the previous ones to see what's been changes.

I really mean it's quite performant from my testing too: my home directory (168G, 924,311 files) only takes ~9 seconds to scan with rclone lsl on my machine, and I expect most directories that would be scanned will be much smaller. My ~/Documents directory (57G, 148,873 files) only takes 1.5 seconds to scan, and pairing that with a Calculating differences... message like @loveisfoss mentioned sounds like a pretty good solution to me.

Pkcarreno commented 11 months ago

It's expected that the syncing loop is still happening in the latest version (8.1)?

hwittenborn commented 11 months ago

It appears so @Pkcarreno, see https://github.com/hwittenborn/celeste/issues/168 for the issue on that.