martymac
commented
3 years ago Is there a way to throttle the transfer rate for fpsync to limit it to 1.25 Gbps. Running 16 jobs using the following syntax limit sthe rate to 1.25 Gbps in a test dataset comprising of 10 MB files:
fpsync -n 16 -s 37772160 /srcdir /targetdir
This command will not enforce any bandwidth limit. It will start 16 local rsync jobs in parallel that will synchronize data as fast as they can.
Option -s will create buckets of 37772160 bytes max per job (it does not limit bandwidth, only the max size of each job's data to sync).
Any insights how fpsync is determining the transfer rate and if there is a better way to calculate the job size to throttle the overall transfer rate?
You will be able to limit per-job transfer rate by using dedicated rsync option --bwlimit and using fpsync's -o switch to pass it to rsync, something like :
$ fpsync -n 16 -s 37772160 -o "-lptgoD -v --numeric-ids --bwlimit=xxx" /srcdir /targetdir
Where xxx is the max transfer rate you want for a single job (so the overall max transfer rate will be 16 * xxx). See also rsync(1) for --bwlimit option.
Hope this helps, Best regards,
Is there a way to throttle the transfer rate for fpsync to limit it to 1.25 Gbps. Running 16 jobs using the following syntax limit sthe rate to 1.25 Gbps in a test dataset comprising of 10 MB files:
fpsync -n 16 -s 37772160 /srcdir /targetdir
However, 16 jobs x 36 MiB (37772160 bytes) should transfer ~576 MiB/s(which is ~ 4.12 Gbps). How is fpsync restricting this to 1.25 Gbps?
Another test with different dataset shows using the same command drives ~6 Gbps transfer rate.
Any insights how fpsync is determining the transfer rate and if there is a better way to calculate the job size to throttle the overall transfer rate?