search

mandarons / icloud-drive-docker

Dockerized iCloud Client - make a local copy of your iCloud documents and photos, and keep it automatically up-to-date.
BSD 3-Clause "New" or "Revised" License
962 stars 46 forks source link

Continue sync where last stopped #195

Closed scifi1 closed 4 months ago

scifi1 commented 4 months ago

Use case Would it be possible to implement a way for the application to restart sync where it last stopped? I have 60 000+ files and one complete sync pass takes days (detecting and skipping one file consumes about 2 seconds)

I suppose it needs to be solution based on looping around a sequence of filters to chop up the complete sync task. Maybe then restart sync using the last filter.

mandarons commented 4 months ago

Interaction with iCloud is polling based. The app needs to traverse the directory tree from its root to leaves to detect changes. Resuming the sync from where it stopped (due to error, timeout etc.) will only help with that iteration of tree traversal. It will not speed up next iteration as it must traverse the whole tree to detect changes.

However, current approach is serial - process one path and then move on to the next path. I am working on next-gen of this app which will be utilizing concurrency to significantly improve the performance.

scifi1 commented 4 months ago

Hi, that would be awesome and worth waiting for!

Would it be possible to also add a feature such as a "one time" sync option suitable for cron-jobs?

mandarons commented 4 months ago

One time sync is currently available. Just set sync_interval to -1 to exit after sync.

scifi1 commented 4 months ago

Cool, thanks!

scifi1 commented 4 months ago

Thank you

lostb1t commented 3 months ago

@mandarons is there a branch available for the work going into concurrency?

scifi1 commented 3 months ago

@mandarons is there a branch available for the work going into concurrency?

I thought I would look into spinning up a bunch of containers, with the obvious drawback of multiple config files with different path expressions....Concurrency support would really be good..

mandarons commented 3 months ago

Concurrency work hasn't started yet for this project. I am currently working on making the library that is used by this app, concurrent aioicloud.