search

digint / btrbk

Tool for creating snapshots and remote backups of btrfs subvolumes
https://digint.ch/btrbk/
GNU General Public License v3.0
1.71k stars 124 forks source link

rate limits are not respected #386

Open Draic opened 3 years ago

Draic commented 3 years ago

I have set up a rate limit in my btrbk config to reduce strain on the network when a backup is running, but the rate limit is not respected and btrbk is still maxing out my internet upstream (~11Mbit/s).

I added 

rate_limit              4m

to the btrbk.conf but it is using more than double that limit. Is there anything I must do before the rate limit works?

digint commented 3 years ago

If rate_limit is set, btrbk pipes the ssh traffic through mbuffer -r <limit> (on the host runing btrbk). This used to work just fine, maybe you are using an old / buggy version of mbuffer?

You can see the commands used by running:

# btrbk run -n -l debug  2>&1 | grep '###'

Another decent way to debug this is to run btrbk --progress, which will display the output of mbuffer along with some stats on the console.

Please also try to configure rate_limit_remote instead of rate_limit. This will run mbuffer on the remote side.

Draic commented 3 years ago

the mbuffer is getting the right command: ssh -i /etc/btrbk/ssh/id_ed25519 -o compression=no root@sprawl xz -d -c | mbuffer -v 1 -q -m 256m -r 4m | btrfs receive /srv/dev-disk-by-label-array/@backup/ running the process with the --progress flag also claims that the limit is respected, but my overall network bandwith will still max out when I run this.

digint commented 3 years ago

I did not get around to test this by now, do you have any progress on this? Did you try most recent version of mbuffer?

Adding "upstream bug" label for now: after your last post it's obvious that btrbk runs the right commands.

Draic commented 3 years ago

I do not really have an understanding how mbuffer works and how it is implemented withing BTRBK, but I can see that the command calls it as it should. But despite having a set limit, it gets exceeded during the backup process regularly. The rate IS lower, but the limit isn't strictly adhered to, so it is pointless if you need it to make sure btrbk will not hinder other operations on the network. Even with the limit set you can see your network traffic to max out every few seconds. As I have not found a solution I begrudgingly changed my setup to run BTRBK by night, when it doesn't matter as much.