[question] regarding nested subvolumes support and send/receive

[zilexa]

Not a bug, just questions:

Please verify the following: If I understand correctly, you do not use BtrFS send/receive. A "snapshot" created by your tool:

1. can be created with a regular folder as source, it does not have to be a BtrFS subvolume (but the disk filesystem needs to be BtrFS).
2. can have a regular folder as destination, the snapshots/backups you create are not subvolumes and don't have to be on a subvolume.
3. you are not dependent on subvolumes at all.
4. you do not use BtrFS send/receive.
5. This also means the timeline backups can take up a lot of space, as these are not BtrFS snapshots?

Question 2: Do you think the following is possible? I am considering to only use a single subvolume per disk. The single subvolume directory structure looks like this: / is the single subvolume and it contains 2 regular subfolders:

/Users/user1/.../.../... /Users/user2/.../.../... /Users/user3/.../.../... /Media/incoming/.../.../... /Media/shows/.../.../... /Media/movies/.../...

I would like to have a destination snaps folder (that will be nested subvolumes) inside each UserX folder. Each snapshot is a snapshot of UserX stored in /Users/UserX/snaps/2021/... for each user.

[zilexa]

Sorry I completely changed my questions 3 times now, as I now finally have a better understanding of BtrFS, its native snapshot and send/receive backup capabilities AND the benefits of your tool. The questions above remain to verify I understand your tool correctly. Thank you in advance for your help!

[3coma3]

Hi @zilexa ! Sorry for the delayed response.

Your assumptions are correct, this tool is intended to backup non-btrfs filesystems especially, using a btrfs filesystem for the snapshot storage. As the source filesystem is solely read by rsync, everything is read without regards for btrfs subvolumes (which would appear as standard subdirectories).

The source filesystem could, however, be btrfs. For that case I think the ideal would be to revert to btrfs send/receive, but support for this feature isn't there yet. In that case the semantics regarding subvolumes would depend on send/receive.

So regarding your points:

Question 1

1. It does use any regular folder as source, and it does not have to be a BtrFS subvolume. The source disk filesystem does not need to be BtrFS, while the destination storage needs to be BtrFS.
2. It uses a regular folder as destination, but it needs to be a BtrFS destination (the destination folder should be under, or be the root of a BtrFS volume). The snapshots/backups you create are subvolumes and may be on a subvolume.
3. False. The system needs a subvolume for the leafs (which are the roots of each snapshot).
4. True
5. False, destinations are subvolumes which are based on the last snapshot taken. That's how it leverages the BtrFS COW capabilities so it doesn't take a lot of space (of course this depends on how much the snapshots change between each other, but that's not specific to this system but to COW in general).

Question 2

I think that again, by using a standard rsync read, anything you present as the source filesystem will be copied "normally" (it's rsync after all). So if you use some kind of unionfs the same rules will apply. This is intentional.

The goal with this script was originally to make it very general and simple, which in turn (not without irony) paves the way to add all kinds of features, extensions and specific thingies. So special behaviours with specific source filesystems may be added in the future, I think it would be cool. As long as it keeps being compact, understandable and extensible. So far it manages to keep doing that so it's all good to try out new stuff whenever anyone wants to, just drop a PR and we can test it!

Thanks for the questions, they might come in handy for a FAQ in the future.

Cheers

[zilexa]

Thanks a whole lot. I did read about the leaves being actual Snapshots, should have realized that. But this helps.

I understand bcachefs is getting very close to be mainlined, perhaps later this year we have to figure that FS out.

For now, I have to make a few choices, as I don't want realtime mirroring (BtrFS raid) and I do want tiered caching to ensure the SSD is always used/HDDs sleep most of the time (via MergerFS), and I want subvolumes per user, it is quite complicated to find a way to protect against disk failure as SnapRAID ("nightly raid1") only works with a single subvolume per disk.

If you would support send/receive, that would allow me to simply use it to create the timeline backup and the snapshot subvolume leaves on a disk and use send/receive to send all of it to a secondary disk. Both not using rsync to keep it efficient. I will close this one for now, thanks again!