search

cjnaz / rclonesync-V2

A Bidirectional Cloud Sync Utility using rclone
MIT License
355 stars 39 forks source link

Initializing large folders #26

Closed leviathan747 closed 5 years ago

leviathan747 commented 5 years ago

I have a folder I am trying to sync (~200-300MB). Running this tool with --first-sync was painfully slow. I attempted to do an "rclone sync" manually and then point the script at that folder but ran into errors. Is there a recommended way to get started with large folders?

cjnaz commented 5 years ago

A couple benchmarks...

I sync about 500MB nightly with Dropbox. Here a run with 86 files copied down from Dropbox which ran for about 30 minutes, plus or minus. 

2019-04-01 03:00:01,171:  ***** BiDirectional Sync for Cloud Services using rclone *****
2019-04-01 03:00:01,184:  Synching Path1  </mnt/<path>/Dropbox/>  with Path2  <Dropbox:>
2019-04-01 03:00:30,147:     154 file change(s) on Path2:   23 new,    0 newer,    0 older,  131 deleted
2019-04-01 03:01:36,102:  >>>>> Successful run.  All done.

2019-04-01 03:30:01,923:  ***** BiDirectional Sync for Cloud Services using rclone *****
2019-04-01 03:30:01,937:  Synching Path1  </mnt/<path>/Dropbox/>  with Path2  <Dropbox:>
2019-04-01 03:30:28,072:      63 file change(s) on Path2:   63 new,    0 newer,    0 older,    0 deleted
2019-04-01 03:32:28,191:  >>>>> Successful run.  All done.

I've since changed to OwnCloud. Here's a run with 118 files copied out of OwnCloud (local) to a local disk that ran about 5 minutes - probably much less. I have the sync running every 5 minutes via cron, and getting all of the files extracted runs 2 to 3 5-minute cycles. The time is probably dominated by the FolderSync push from my phone which runs 5 to 7 minutes.

2019-06-03 05:34:01,251:  ***** BiDirectional Sync for Cloud Services using rclone *****
2019-06-03 05:34:01,274:  Synching Path1  </mnt/<path>/owncloud/>  with Path2  <owncloud:>
2019-06-03 05:34:02,137:     225 file change(s) on Path2:   97 new,    0 newer,    0 older,  128 deleted
2019-06-03 05:34:32,940:  >>>>> Successful run.  All done.

2019-06-03 05:39:01,635:  ***** BiDirectional Sync for Cloud Services using rclone *****
2019-06-03 05:39:01,657:  Synching Path1  </mnt/<path>/owncloud/>  with Path2  <owncloud:>
2019-06-03 05:39:02,489:      21 file change(s) on Path2:   21 new,    0 newer,    0 older,    0 deleted
2019-06-03 05:39:09,564:  >>>>> Successful run.  All done.
cjnaz commented 5 years ago

Internally, Path1 may be considered the master, and deltas on Path2 are individually applied to Path1, and finally rclone sync is used to update Path2 to match (assuming there were other changes on Path1). Functionally, Path1 and Path2 will be made to match, so their order does not matter in the end. If you have a lot of files to push from place-A to place-B then making place-A be Path1 will (should) be most efficient because rclone sync may run concurrent copies/operations.

Doing an initial rclone sync should be effective, followed by a rclonesync --first-sync. What error did you get (run with --verbose)?

leviathan747 commented 5 years ago

Switching Path1 and Path2 seemed to do the trick.