rclone / rclone

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
https://rclone.org
MIT License
46.81k stars 4.18k forks source link

bwlimit schedule not working next day #1268

Closed mechanimal82 closed 6 years ago

mechanimal82 commented 7 years ago

So I'm running rclone on my QNAP NAS to upload to Amazon encrypted. As we are on only an OK connection I'm limiting bandwidth during the day to ensure that our download speed is not affected (when using the upload without limit affects our download speed due to the inability to properly handshake with the server)... Anyway; I set the schedule as

/usr/bin/rclone copy /share/CACHEDEV1_DATA/Multimedia/TV/ acdcrypt:/TV/ --bwlimit "07:00,1.75M 23:30,off" -c --no-traverse --delete-after -vv --log-file=/share/CACHEDEV1_DATA/Public/uploadTV.log

Below are extracts of the log which are relevant... Happy to attach the entire log if required.

Essentially it seems that the limit intially works OK when the scedule day is the same day the command was run (as noted with the scedule correctly changing at 23:00)... when the new day comes rclone seems to request a limit set (as seen in the log), but this does not get passed properly.

Am I missing something? Seems like a bug to me unless my lack of experience is yet again failing me.

2017/03/22 19:37:31 INFO  : Starting bandwidth limiter at 1.750MBytes/s
2017/03/22 19:37:31 DEBUG : rclone: Version "v1.36" starting with parameters ["/usr/bin/rclone" "copy" "/share/CACHEDEV1_DATA/Multimedia/TV/" "acdcrypt:/TV/" "--bwlimit" "07:00,1.75M 23:30,off" "-c" "--no-traverse" "--delete-after" "-vv" "--log-file=/share/CACHEDEV1_DATA/Public/uploadTV.log"]

At 23:00, as scheduled the limits are disabled and my speed goes to my maximum (2MB):

2017/03/22 23:30:31 NOTICE: Scheduled bandwidth change. Bandwidth limits disabled
2017/03/22 23:30:32 INFO  : 
Transferred:   23.898 GBytes (1.750 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:           44
Elapsed time:     3h53m1s
Transferring:
 * ...Files:

Come 07:00 when the limiter is set to reset back to 1.75MB, it records it correctly:

2017/03/23 07:00:31 NOTICE: Scheduled bandwidth change. Limit set to 1.750MBytes/s
2017/03/23 07:00:32 INFO  : 
Transferred:   80.138 GBytes (2.002 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:           95
Elapsed time:    11h23m1s
Transferring:
 * ...Files:

2017/03/23 07:01:32 INFO  : 
Transferred:   80.240 GBytes (2.002 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:           95
Elapsed time:    11h24m1s
Transferring:
 * ...Files:

However it appears the schedule has not reset to my required 1.75MB:

2017/03/23 07:57:32 INFO  : 
Transferred:   85.983 GBytes (1.983 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:          100
Elapsed time:    12h20m1s
Transferring:
 * ...Files:

2017/03/23 07:58:32 INFO  : 
Transferred:   86.085 GBytes (1.983 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:          100
Elapsed time:    12h21m1s
Transferring:
 * ...Files:

2017/03/23 07:59:32 INFO  : 
Transferred:   86.188 GBytes (1.982 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:          101
Elapsed time:  12h22m1.1s
Transferring:
 * ...Files:
mazeprince commented 7 years ago

It works as expected! You are just confused by the average speed on the total transferred. At 07:00:32 you had transferred 80,138 GB, at 07:57:32 it was 85.983 GB. So: (85983-80138)/(5760)=1,7091 MB / s The output doesn't look that way because you transferred 80.138 GB - 23.898 GB = 56,24 GB at full speed. So just for a complete calculation: (23,8 1,75 + 56,3 2 + 5,8 1,75) / 85,9 = 1,91 MB / s The difference to the logfile is probably because your full speed is a little bit higher than 2 MB / s.

Cheers

edit:to be sure you can always check your current speed, for example with nload under linux.

mechanimal82 commented 7 years ago

Ah - That's average speed. OK; makes a little more sense.

Only I also checked the network speed from the resource monitor on the QNAP and it's outputting a constant 2MBps, indicating that the process is still going full pelt.

Let me try again today and leave it running a little longer tomorrow and hopefully it'll return back - else I'll have my wife complaining that the Internet is going slow again :-)

Thanks - At least I know it's not something I'm doing wrong; just need to be a little more patient.

mazeprince commented 7 years ago

Yeah, another thing: The limit isn't really a "fixed" limit, like a complete cap or something. When i look at the current speed on my server it very often goes over the limit. But in the end the average speed (lets say average over a few minutes) equals the bwlimit.

ncw commented 6 years ago

SInce we haven't heard back, I'm going to assume this is fixed!