Feature Request: Calendar Suffix Customisation

[dcrdev]

I know it's been discussed many times before but could it be considered once more...

Would it be possible to allow customisation of the snapshot calendar based suffix via policy i.e. a user could define _weekly as _other in the config file.

Specifically why I would find this useful is integration with the vfs_shadow_copy2 module in samba, if I could define these tags as arbitrary numbers via a config, then I could trick the shadow:format directive into recognising all snapshots instead of just one set.

I think you've mentioned that you're reluctant to do this due to the shadow:format directive not being implemented in some distros. I've personally not dealt with a distro in a long time where this feature isn't supported - it's supported in the current versions of rhel, debian and opensuse.

[jimsalterjrs]

You're welcome to submit a PR, but I currently don't have the bandwidth to tackle this pro bono.

[jasonblewis]

I suspect one reason it's tricky to implement is that it makes working out what to expire more difficult.

however one option might be to move some info about the snapshot into a property on the snapshot using the zfs snapshot -o sanoid:snapshotclass=daily or something. This gets stored with the snapshot and can be queried later. See man zfs snapshot for details, also the User Properties section of the man page.

[phreaker0]

@dcrdev, @jasonblewis recent version of vfs_shadow_copy2 have the new option "shadow:snapprefix" exactly for that.

[dcrdev]

@phreaker0 doesn't that option only work on the prefix though? Sanoid generates snapshots in this format: autosnap%Y-%m-%d%H:%M:%S_hourly

[phreaker0]

@dcrdev seems you are right :(, such a bummer

[gnordli]

Windows previous versions through samba work fine with this configuration:

   shadow: snapdir = .zfs/snapshot
   shadow: sort = desc
   shadow: format = _%Y-%m-%d_%H:%M:%S
   shadow: snapprefix = ^autosnap
   shadow: delimiter = _
   shadow:localtime = yes

The only issue I see is when you have multiple backup types (daily/weekly/etc....) they will sometimes have the same name because the suffix is stripped and only the time format remains.

Edited the comment, Sorry, didn't notice the formatting ate the leading _ on the original post.

[phreaker0]

@gnordli nice, I think this is the best way to solve the described problem.

[darkpixel]

Sorry @gnordli, I ignored the part about having multiple snapshot types. None-the-less, I think this should remain open and support for multiple snapshots using vfs_shadow_copy2 should be investigated.

I don't think this should be closed.

I've been using shadow_copy2 with zfs-auto-snapshot and just began testing syncoid a few hours ago to see if it's a viable replacement. The good news is, this is my only blocker to production--everything else is great. :)

Anyways, zfs-auto-snapshot has two slight differences that allow it to work with shadowcopy_2.

The first is the ability to set a label like so: /usr/sbin/zfs-auto-snapshot --quiet --syslog --label=05 --keep=30 --recursive tank

Most normal humans would have a label named 'daily', 'hourly', etc...but allowing a customizable label comes in handy.

The other slight difference is where the underscores and dashes are in the snapshot name: zfs-auto-snap_05-2020-03-01-0415

The prefix is zfs-auto-snap_. 05 is the 'type' of snapshot (frequent, hourly, daily, etc...), and the rest is obvious.

That allows me to use a shadow format of zfs-auto-snap_%S-%Y-%m-%d-%H%M. You can sneakily hack around the snapshot type being a number by assigning it to %S which is the number of seconds. This accounts for the multiple types of snapshots and keeps sorting intact.

Anyways, on to why I don't think this should be closed.

Sanoid created snapshots like this: autosnap_2020-02-29_19:53:50_daily. There's an underscore between the date and the time, and also another one between the time and the 'label'.

I believe the example from @gnordli above is wrong (and it didn't work on my system when I tested): shadow: format = %Y-%m-%d%H:%M:%S

It's missing the underscore separating the date and the time.

Unfortunately adding it shadow: format = %Y-%m-%d_%H:%M:%S also didn't work for me.

Am I missing something?

I really wish vfs_shadow_copy3 would get built and just allow matching from a regular expression already. Or maybe zfs could create snapshot entries under .zfs/snapshot with an access, modify, and change time of the snapshot date. ;)

[JakobR]

It seems the formatting ate the underscores in @gnordli's comment (if you look closely you can see that part of it is in italics). So it should be shadow:format = _%Y-%m-%d_%H:%M:%S (note the added underscore at the beginning). From my (very limited) testing, it seems to work like this.

[darkpixel]

You are correct @JakobR! I tested and it works on my system. After a bunch of testing, I'm excited to move my systems over to start using sanoid. The two huge benefits for me are the ability to detect the latest snapshot between the local system and remove system and automatically resume, and the ability to resume recursive child datasets. About once a month a Comcast connection or two would go down in the middle of a recursive backup, and other tools didn't support resuming. I'd have to spend hours or days manually bringing the children up to the same snapshot level and then tell the backup program the previous send hadn't really failed....such a huge timesaver!

[shielings]

Vey excited by the @gnordli solution! Do we need to have vfs_catia as mentioned here #345 for this to work? Tried:

zfs set snapdir=visible

vfs objects = shadow_copy2 shadow: snapdir = .zfs/snapshot shadow: sort = desc shadow: format = _%Y-%m-%d_%H:%M:%S shadow: snapprefix = ^autosnap shadow: delimiter = _ shadow:localtime = yes

I suspect its because Windows clients are looking in the /.zfs/snapshot directory and seeing names like ACJ3MA~U instead of autosnap_2020-04-03_09:45:00_frequently

Anyone suggest a fix?

[shielings]

I've fixed it - vfs_catia not required. I needed an extra underscore, after %S. My working config:

vfs objects = shadow_copy2 
shadow: snapdir = .zfs/snapshot
shadow: sort = desc
shadow: format = _%Y-%m-%d_%H:%M:%S_
shadow: snapprefix = ^autosnap
shadow: delimiter = _
shadow: localtime = yes

[abclution]

I just cannot get this to work. I do not understand why. I have zfs-auto-snapshot ones working, but cannot seem to get this working with Sanoid.

Does anyone have this working on Debian Buster?

[darkpixel]

@abclution I do, but I'm not at my desk. I'll try to remember to get it to you tomorrow morning. If I forget, don't hesitate to ping me.

[abclution]

@darkpixel That would be much appreciated. I have been hitting my head against this for many many hours. :(

[darkpixel]

Here's my config from a fileshare:

[corp-compliance]
     comment = Compliance
     path = /tank/corp-compliance
     guest ok = False
     read only = True
     vfs objects = shadow_copy2 full_audit
     available = yes
     hide dot files = yes
     hide files = /.stfolder/
     browseable = yes
     force create mode = 0666
     force directory mode = 0777
     recycle:repository = .recycle/%U
     recycle:keeptree = yes
     recycle:versions = yes
     recycle:touch = yes
     recycle:directory_mode = 0777
     recycle:subdir_mode = 0700
     shadow:snapdir = .zfs/snapshot
     shadow:sort = desc
     shadow:format = _%Y-%m-%d_%H:%M:%S
     shadow:snapprefix = ^autosnap
     shadow:delimiter = _
     shadow:localtime = yes
     acl_xattr:default acl style = everyone
     acl_xattr:ignore system acls = yes
     full_audit:prefix = %I|%u|%m|%S
     full_audit:facility = LOCAL6
     full_audit:priority = ALERT
     full_audit:success = connect disconnect mkdir rmdir rename unlink read write pwrite sendfile fsync ftruncate link mknod pwrite symlink open !readdir !opendir
     full_audit:failure = connect disconnect mkdir rmdir rename unlink read write pwrite sendfile fsync ftruncate link mknod pwrite symlink open !readdir !opendir
     vfs objects = shadow_copy2 full_audit acl_xattr

There's a bit of extra junk in there you probably don't need, but you could probably start with that and tweak until you break something. ;)

[dieechtenilente]

I got my shadow copies working with @shielings config. But I don't exactly know why...

[quotengrote]

@darkpixel

Here's my config from a fileshare:

[...]

There's a bit of extra junk in there you probably don't need, but you could probably start with that and tweak until you break something. ;)

Does this show you all snapshots or only the hourlies? I cant get it to working with all type of snapshots... For german clients i had to set shadow: localtime = no

[darkpixel]

Actually...you're correct. It only shows hourly snapshots on my end. Ugh. Oh well. If users need to roll back more than a few days they can contact IT and I'll manually roll stuff back for them.