search

allanjude / zxfer

A continuation of development on zxfer, a popular script for managing ZFS snapshot replication
BSD 2-Clause "Simplified" License
123 stars 40 forks source link

optimizing get_zfs_list() by running snapshot list commands in the background #65

Open totalAldo opened 5 months ago

totalAldo commented 5 months ago

Hi Allan,

When listing the snapshots on some of my pools, it can take over 15 seconds for each command to finish. One way to reduce the execution time of get_zfs_list() is by running the source and destination zfs list -t snap commands in the background, write the results to temporary files, proceed with other list command and wait for the commands to finish.

Here is an implementation of that logic which has reduced get_zfs_list() execution time by nearly 50%

https://github.com/totalAldo/zxfer/blob/9a7b4e1da5305952863ddbe518403b8c8c18521b/zxfer#L1791-L1919

GernotS commented 5 months ago

Would be great if you could add -w (raw send) to your implementation as well

totalAldo commented 5 months ago

@GernotS I've implemented the -w option in my fork. If you have a chance, please test, and let me know if you encounter any issues.

https://github.com/totalAldo/zxfer/commit/cf589ff25af8be1d9b98b72382c601dacede6d72

GernotS commented 5 months ago

Wow, that was quick, thank you! Your implemantation is surely my favorite now, way faster than the original.

GernotS commented 1 month ago

as your zxfer repo is missing the issues tab I need to write this here. You latest commit uses grep -m which is not supported on Illumos.

thx

totalAldo commented 1 month ago

Hi @GernotS ! Apologies. I have enabled Issues and removed the use of "-m". Would love your feedback if you've enabled the new "-j" option as I've found that to further speed up performance significantly.

GernotS commented 1 month ago

thanks, filed two issues already