search

cjnaz / rclonesync-V2

A Bidirectional Cloud Sync Utility using rclone
MIT License
355 stars 39 forks source link

Improving download performance for many small files #36

Closed kj4tmp closed 4 years ago

kj4tmp commented 4 years ago

First, THANK you for making this great tool. During --first-sync, for small files, I'm seeing a speed of about 0.5 files/s.

Is there a way to improve this performance?

I'm using my own Google drive API key and the command I am currently running to perform the first sync is(after doing a little googling): ~/.scripts/rclonesync.py ~/Drive Drive: --first-sync --filters-file ~/.rclonesyncwd/Filters.txt --rclone-args --fast-list --tpslimit=10 --transfers=10 --drive-chunk-size=128M

cjnaz commented 4 years ago

Unfortunately, rclonesync only issues one rclone command at a time, so the parallelism supported natively by rclone isn't being utilized. The rclone project has a long-standing plan to incorporate a native/built-in bidirectional sync, thus obviating the need for rclonesync (https://github.com/rclone/rclone/issues/2870). You might want to check out https://github.com/ConorWilliams/rsinc/blob/master/rsinc/rsinc.py, which I believe supports issuing concurrent rclone commands.

Upgrading rclonesync for concurrent commands is non-trivial surgery, so I do not intend to do this work as I'm assuming rclone will eventually do it better. Anyone is welcome to fork the project and go for it.