search

masc3d / btrfs-sxbackup

Incremental btrfs snapshot backups with push/pull support via SSH
GNU General Public License v2.0
117 stars 20 forks source link

Allow to specify ssh cypher #55

Open s-m-e opened 5 years ago

s-m-e commented 5 years ago

Allowing a user to specify the cypher / encryption implementation in ssh can speed things up considerably in certain scenarios. The idea is that some implementations are accelerated by AES-NI.

In my case, the optimum is ssh -c aes256-gcm@openssh.com. It depends on what is available on both the local and remote machines, varying from Linux distro to Linux distro etc.

masc3d commented 5 years ago

I'd rather avoid to have all sort of ssh options replicated into backup job config.

imho cleaner solution is to actually define a dedicated user and setup preferred ciphers via ssh_config. would you agree?

masc3d commented 5 years ago

also could you please share your measures of regular setup vs optimized for both optimized cipher and compression option? just out of curiosity

s-m-e commented 5 years ago

imho cleaner solution is to actually define a dedicated user and setup preferred ciphers via ssh_config. would you agree?

From a design perspective, yes, it's cleaner. I agree. In this case, I'd suggest to add this as a recommendation to the manual instead.

also could you please share your measures of regular setup vs optimized for both optimized cipher and compression option? just out of curiosity

1st machine CPU i7-3740QM, 2nd machine CPU Xeon E5-2630L. They are both not exactly the latest models. 1 GBit/s NICs (both Intel) in-between, SSDs (or SSD-cached storage) at both ends. For long-duration loads such as backups, the i7's turbo is disabled (under-designed cooling). With AES-NI, the transfer speed averages close to the theoretical maximum at about 122 MByte/s. Without AES-NI and without turbo, I see drops to 40 MBytes/s and averages of about 80 MByte/s. Without AES-NI but with activated turbo, it is somewhere around 110 MByte/s (before thermal issues slow things down).

masc3d commented 5 years ago

From a design perspective, yes, it's cleaner. I agree. In this case, I'd suggest to add this as a recommendation to the manual instead.

alright

I see drops to 40 MBytes/s and averages of about 80 MByte/s. Without AES-NI but with activated turbo, it is somewhere around 110 MByte/s (before thermal issues slow things down).

would be more curious about absolute numbers of transmission duration in comparison, as I have seen btrfs send / receive not being terribly efficient in general, so is possible that those drops are actually neglectable considering overall backup runtime.

but also it's been quite a while i looked at it in detail and btrfs may have improved in this respect since then.

s-m-e commented 5 years ago

btrfs send / receive not being terribly efficient in general

Yes, there are apparently still issues. I am trying to work my way through that. I should add that the above numbers are for random data streams simply read from files and, besides, zfs send/receive - not yet btrfs send/receive on its own. The latter still falls back, but I have yet not worked my way though optimization options. One bottleneck at a time - and ssh encryption is a significant one.