Incompatible snapshot in backup chain local -> local -> remote

[diresi]

So I have 3 zfs pools. "smol" and "backup" on machine A and another one named "big" on machine B.

I ran initial backups without errors, first backing up "smol" to "backup" locally on A and then pushing out "backup" from A to "big" on B:

A/smol -> A/backup A/backup -> B/big

This worked fine. Next day I'm playing around but backup runs show errors. This one is on machine A where I try to run a local backup, when it suddenly complains about a snapshot of the other run (remote backup to B), a snapshot I thought it would ignore completely:

$ zfs-autobackup -v local backup/autobackup 
  zfs-autobackup v3.2 - (c)2022 E.H.Eefting (edwin@datux.nl)

  NOTE: Source and target are on the same host, excluding target-path from selection.

  Selecting dataset property : autobackup:local
  Snapshot format            : local-%Y%m%d%H%M%S
  Timezone                   : Local
  Hold name                  : zfs_autobackup:local

  #### Source settings
  [Source] Keep the last 10 snapshots.
  [Source] Keep every 1 day, delete after 1 week.
  [Source] Keep every 1 week, delete after 1 month.
  [Source] Keep every 1 month, delete after 1 year.

  #### Selecting
  ...
  [Source] smol/HOMES/data/etc: sending to backup/autobackup/smol/HOMES/data/etc
! [Target] backup/autobackup/smol/HOMES/data/etc@B-20240310205801: Incompatible snapshot
! [Source] smol/HOMES/data/etc: FAILED: Please destroy incompatible snapshots or use --destroy-incompatible.
  ...

This is unexpected. Checking zfs props I can see the mentioned snapshot is properly labeled as autobackup:B but considered nevertheless. I can only assume that destroying the snapshot would break remote backups of that pool.

As far as I understand the whole idea is that only matching snapshots should be considered for all operations (backing up and thinning), in this case that would be the ones with "autobackup:local" props, which doesn't exist here.

[diresi]

Are such backup chains even supported? On other datasets on the intermediary "backup" pool I can see errors due to snapshots being written, where I can only guess that the only change is newer snapshots from the other backup to remote pools.

[psy0rz]

yes this setup is possible, aslong as there are absolutely no changes made on A/backup.

in your case the snapshot is incompatible because it contains changes.

try making the target readonly to prevent changes.

also see https://github.com/psy0rz/zfs_autobackup/wiki/Mounting

as for the current situation, you probably need to --destroy-incompatible and do a resend of the dataset from A/backup -> B/big (e.g. run that backup also with --destroy-incompatible)

[diresi]

Thanks for replying, and the docs about mounting. This might be the reason for modifications as I played around with mountpoints as well. And now that that's on the table I remember that I had to unset readonly to let zfs create mountpoints of nested datasets.

Seems that I broke it... I'll try again and will update/close this issue then.

Are there best practices regarding backups of nested datasets? And maybe about having backups mounted (as readonly) for easy access?

[Scrin]

Ultimately the appropriate best practices depend on your needs, but in general you'll want to either mount the datasets read-only or not mount them at all (my preference). For accessing the contents without zfs sending them straight away, or tinkering with backups and/or their snapshots in general, I always recommend working with clones rather than the "actual" backup snapshots. I outlined some of this in another thread a while back here: https://github.com/psy0rz/zfs_autobackup/discussions/201#discussioncomment-6403422

[diresi]

I see, thanks for that link as well! I'll try to remember clones, a zfs feature I never used so far (shame on me!).

I'd like to get backups mounted, though, to quickly browse through snapshots. I think I broke things by creating those mountpoints after the fact and having backed up nested datasets, which seems overly complicated for my needs. I guess I'm gonna try flat datasets (not mountpoints) under common roots, which aren't backed up. This way snapshots and backups remain isolated and chances of me breaking a whole subtree (well, tree in my case) seem less. The autobackup:xyz=child property seems perfect for that.

To explain my trail of thoughts: I use dataset layouts that I found promoted on the internet looking like pool/TOPIC/root/dataset, in my case this is smol/HOMES/data/docs, smol/HOMES/data/photos, and so on. There's no system data (like / or /boot) nor vm images or any of that. I just like to keep my data as long as possible :).

Anyway, I think it's safe to close this issue again. It was my own fault due to not understanding the consequences of my own actions. Thanks for helping me!

[psy0rz]

no problem :)

thanks @Scrin