search

jimsalterjrs / sanoid

These are policy-driven snapshot management and replication tools which use OpenZFS for underlying next-gen storage. (Btrfs support plans are shelved unless and until btrfs becomes reliable.)
http://www.openoid.net/products/
GNU General Public License v3.0
3.09k stars 301 forks source link

Parallel pruning #935

Open nijave opened 3 months ago

nijave commented 3 months ago

Has parallel pruning ever come up? It's significantly faster for me (almost the same amount of time to prune 100s of snapshots in parallel as it takes to prune a single snapshot)

Usually I do

zfs list -t snapshot -o name my_pool | grep <some criteria> | while read -r snap; do zfs destroy "$snap" & done

Batching might be a middle ground (a 10s or 100s at a time). This would greatly help when reducing retention rules

Looks like it took ~4-5 minutes to prune 1500 snapshots across 2 pools running all in parallel vs ~10 seconds a piece when running sanoid --snapshots-prune

jimsalterjrs commented 3 months ago

Has parallel pruning ever come up? It's significantly faster for me

Interesting! No, hadn't really considered it. I've had to delete thousands of unwanted snapshots from a system before but never tried massive parallelism; I would do your loop with ; done instead of & done.