zfsonlinux / zfs-auto-snapshot

ZFS Automatic Snapshot Service for Linux
GNU General Public License v2.0
840 stars 242 forks source link

Add a switch for the `zfs list` invocation. #16

Open dajhorn opened 11 years ago

dajhorn commented 11 years ago

Pursuant to zfsonlinux/zfs#450, the slow return of some zfs list invocations is arguably a bug. The merge in zfsonlinux/zfs-auto-snapshot#15 kludges the zfs list invocation to improve performance.

Provide a switch to allow the user to choose between correctness or performance. If corner-cases are discovered in the kludge, then add a --fast switch. Otherwise, add a --slow switch to preserve the original code until performance improves.

bill-mcgonigle commented 9 years ago

This is great! Slow snapshot listings (really the high load associated with) have been my #1 headache with running frequent snapshots (which are super-handy).

System #1: 
# time zfs list -H -t snapshot -S creation -o name | wc -l
1686

real    0m32.509s
user    0m0.208s
sys     0m0.624s
# time zfs list -H -t snapshot -o name -s name|grep zfs-auto-snap |awk '{ print substr( $0, length($0) - 14, length($0) ) " " $0}' |sort -r -k1,1 -k2,2|awk '{ print substr( $0, 17, length($0) )}' | wc -l
1686

real    0m0.130s
user    0m0.048s
sys     0m0.075s

System #2:

# time zfs list -H -t snapshot -S creation -o name | wc -l
3755

real    5m8.268s
user    0m1.039s
sys     0m13.140s
# time zfs list -H -t snapshot -o name -s name|grep zfs-auto-snap |awk '{ print substr( $0, length($0) - 14, length($0) ) " " $0}' |sort -r -k1,1 -k2,2|awk '{ print substr( $0, 17, length($0) )}' | wc -l
3755

real    0m4.709s
user    0m0.237s
sys     0m0.845s

I've edited my crontabs and no errors so far (just a few frequents of testing), but I can't imagine switching back unless something unexpected happens.

devZer0 commented 7 years ago

further performance optimization is possible, if zfs-auto-snapshot would not pull a whole list of all snapshots but only the list of the appropriate filesystem (or sub-fs) to be snapshotted. that may (depending of use case) skip most of the snapshots ( see https://github.com/zfsonlinux/zfs-auto-snapshot/issues/62 )

  1. pull whole list of ALL snapshots

    while true; do (time zfs list -r -H -t snapshot -S creation -o name|wc -l) 2>&1 |grep real;sleep 60 ;done

    real 7m49.567s

  2. pull whole list in a more efficient way (--fast option)

    while true; do (time zfs list -r -H -t snapshot -o name -s name) 2>&1 |grep real;sleep 60 ;done

    real 0m3.978s real 0m4.359s real 0m6.028s real 0m6.199s real 0m5.385s real 0m8.575s

  3. pull only the sub-fs the traditional way. mind that this also speeds up things a LOT if you have a lot of snapshots

    while true; do (time zfs list -r -H -t snapshot -S creation -o name zfspool/backup/adminstation.local|wc -l) 2>&1 |grep real;sleep 60 ;done

    real 0m4.984s real 0m5.084s real 0m5.631s real 0m6.671s real 0m8.397s real 0m5.398s

  4. (fastest) pull only sub-fs in a more efficient way (let`s combine --fast + pull only sub-fs list)

    while true; do (time zfs list -r -H -t snapshot -o name -s name zfspool/backup/adminstation.local|wc -l) 2>&1 |grep real;sleep 60 ;done

    real 0m0.210s real 0m0.058s real 0m0.121s real 0m0.111s real 0m0.024s real 0m0.125s