jimklimov
commented
1 year ago Cheers, do you have any "excluded" datasets? e.g. you snapshot rpool/export and set enabled=off explicitly for rpool/export/home/admin? Some mis-processing like that did happen to me as (hopefully) fixed by #592. Can't say I saw problems with thinning per se, but that code did not support deletion of many different datasets' snapshots other than by recursion from their ancestor.
Another possible cause that comes to mind is the shell command-line length limit (often circa 5000 chars) - if you have many snapshots to clean away at once (in a comma-separated list recursing from e.g. rpool/export@znapzend-1,znapzend-2,...,znapzend-1000) - this command line might get exceeded and so invalid. It might happen if your backup host was offline for a long enough time to accumulate many 3-minute snapshots for cleanup upon reconnection.
@oetiker : would it make sense to at least log (at best solve) an RFE to fall back to oracleMode or at least re-chunk the cleanup with smaller delete-lists, if there are too many snapshots in the array (and/or generated command line is larger than X, or whatever shell interpreter that system has complains about command line too long)? Maybe just add a configurable max CLI length (per SRC/DST schedule - may have different shells)? So far the issue is theoretical (haven't seen it in practice) but sounds plausible...
yggdr
stale[bot]
znapzend does not seem to thin remote snapshots. E.g. right now, I have configured it to snapshot every 3 minutes and keep 12 hours worth of them on my backup machine (in addition to hourly and daily ones). Since it is 16:30 now, I'd assume to see 3min-snaps until around 4 in the morning, but they go back until midnight. And when I scroll through the list of snaps, I see thousends of snapshots 3 min apart starting from 3-4 days ago going back months!
With a plan of
dst_nas_plan=12hours=>3minutes,36hours=>1hours,2weeks=>1days,3months=>1weeks,2years=>1monthsI should see at most 12hours * (60min/hour) / (3min/snapshot) = 240 snaps every 3 min, add to that 36 hourly, 14 daily, ca. 12 weekly, and 24 monthly and we arrive at less than 330 snapshots per dataset. Yet an individual dataset on the backup machine has 105000!Running as
znapzend --debug --autoCreation --cleanOffline, the user znapzend uses to connect to the remote has permissionscreate,destroy,hold,mount,receive,send,snapshot,userprop.edit: Just for the hell if it I tested out running znapzend with --features=oracleMode, it seems to now (slowly, but surely) destroy the old snapshots. I suppose the bug lies in the fact that when so many snapshots accumulate (and I still don't know why they did in the first place), trying to delete them all at once in one long command line breaks some limit. Batching them in sets should solve that.