Errors when using subvolumes

[mwt]

I have a subvolume at /, /home, and /usr. When I run the script:

> ./btrfs-auto-snapshot -n /
ERROR: cannot find real path for '/@': No such file or directory
ERROR: cannot find real path for '/@home': No such file or directory
ERROR: cannot find real path for '/usr/@': No such file or directory
ERROR: cannot find real path for '/usr/@/usr': No such file or directory
ERROR: cannot find real path for '/usr/@home': No such file or directory
ERROR: cannot find real path for '/home/@': No such file or directory
ERROR: cannot find real path for '/home/@/usr': No such file or directory
ERROR: cannot find real path for '/home/@home': No such file or directory
Doing snapshots of /
mkdir //.btrfs
Notice: btrfs subvolume snapshot / /.btrfs/btrfs-auto-snap_-2022-07-07-2121

If I try to backup everything

> ./btrfs-auto-snapshot -n //
ERROR: cannot find real path for '/@': No such file or directory
ERROR: cannot find real path for '/@home': No such file or directory
ERROR: cannot find real path for '/usr/@': No such file or directory
ERROR: cannot find real path for '/usr/@/usr': No such file or directory
ERROR: cannot find real path for '/usr/@home': No such file or directory
ERROR: cannot find real path for '/home/@': No such file or directory
ERROR: cannot find real path for '/home/@/usr': No such file or directory
ERROR: cannot find real path for '/home/@home': No such file or directory
Doing snapshots of / /usr /usr /home
mkdir //.btrfs
Notice: btrfs subvolume snapshot / /.btrfs/btrfs-auto-snap_-2022-07-07-2122
mkdir /usr/.btrfs
Notice: btrfs subvolume snapshot /usr /usr/.btrfs/btrfs-auto-snap_-2022-07-07-2122
mkdir /usr/.btrfs
Notice: btrfs subvolume snapshot /usr /usr/.btrfs/btrfs-auto-snap_-2022-07-07-2122
mkdir /home/.btrfs
Notice: btrfs subvolume snapshot /home /home/.btrfs/btrfs-auto-snap_-2022-07-07-2122

Snapshots themselves mostly seem to work. Though, keep isn't reliably deleting the old backups.

[hunleyd]

thanks for the report @mwt . Sorry to hear it's not working for you. I assume you're using the 2.0 release?

[mwt]

Actually, I downloaded directly from the branch. I'll try again with the release.

[hunleyd]

thoughts @ams-tschoening ? fwiw, i'm seeing this as well here now:

sudo ./btrfs-auto-snapshot --dry-run //
./btrfs-auto-snapshot: line 468: [: : integer expression expected
ERROR: cannot find real path for '/@': No such file or directory
ERROR: cannot find real path for '/@home': No such file or directory
ERROR: cannot find real path for '/@snapshots': No such file or directory
ERROR: cannot find real path for '/home/@': No such file or directory
ERROR: cannot find real path for '/home/@home': No such file or directory
ERROR: cannot find real path for '/home/@snapshots': No such file or directory
ERROR: cannot find real path for '/home/@/srv': No such file or directory
ERROR: cannot find real path for '/home/@/var/lib/portables': No such file or directory
ERROR: cannot find real path for '/home/@/var/lib/machines': No such file or directory
Doing snapshots of / /srv /var/lib/portables /var/lib/machines /home
mkdir //.btrfs
Notice: btrfs subvolume snapshot / /.btrfs/btrfs-auto-snap_adhoc_2022-07-09-1533
mkdir /srv/.btrfs
Notice: btrfs subvolume snapshot /srv /srv/.btrfs/btrfs-auto-snap_adhoc_2022-07-09-1533
mkdir /var/lib/portables/.btrfs
Notice: btrfs subvolume snapshot /var/lib/portables /var/lib/portables/.btrfs/btrfs-auto-snap_adhoc_2022-07-09-1533
mkdir /var/lib/machines/.btrfs
Notice: btrfs subvolume snapshot /var/lib/machines /var/lib/machines/.btrfs/btrfs-auto-snap_adhoc_2022-07-09-1533
mkdir /home/.btrfs
Notice: btrfs subvolume snapshot /home /home/.btrfs/btrfs-auto-snap_adhoc_2022-07-09-1533

[mwt]

Yeah, I get it on the version too. I get the integer expected when using the keep option. I didn't think I get it without using that option.

[hunleyd]

@mwt can you provide me the output of btrfs subvolume list / ?

[mwt]

@mwt can you provide me the output of btrfs subvolume list / ?

ID 256 gen 6689 top level 5 path @
ID 258 gen 6689 top level 256 path usr
ID 259 gen 6689 top level 5 path @home

[hunleyd]

@mwt can you provide me the output of btrfs subvolume list / ?

ID 256 gen 6689 top level 5 path @
ID 258 gen 6689 top level 256 path usr
ID 259 gen 6689 top level 5 path @home

Thanks! So @ is / and @home is /home (and both are "top-level" subvolumes (id 5)) and usr (/usr) is a 'normal' subvolume. Snapshotting @ does not include the contents of /usr. When you run btrfs-auto-snapshot // are you wanting both @ and @home and /usr ? Or just @ and @home

[mwt]

@mwt can you provide me the output of btrfs subvolume list / ?

ID 256 gen 6689 top level 5 path @
ID 258 gen 6689 top level 256 path usr
ID 259 gen 6689 top level 5 path @home

Thanks! So @ is / and @home is /home (and both are "top-level" subvolumes (id 5)) and usr (/usr) is a 'normal' subvolume. Snapshotting @ does not include the contents of /usr. When you run btrfs-auto-snapshot // are you wanting both @ and @home and /usr ? Or just @ and @home

I just want to back up @ by itself without including any of the subvolumes. I would be ok with anything though. The problem is that these errors are generated no matter what I try to backup.

I don't really want //. I want /. I just used // as an example of something that should work.

[ams-tschoening]

./btrfs-auto-snapshot: line 468: [: : integer expression expected

That line doesn't even exist in my version, and seems to have been introduced later. In MASTER it's the following currently, which doesn't make too much sense to me right now:

writeable="${cmdline[writable]}"

https://github.com/hunleyd/btrfs-auto-snapshot/blob/ba1389285d2af4afb783911f54de89372c62481f/btrfs-auto-snapshot#L468

So please both of you link the exact commit hashes of the versions you tested, otherwise debugging seems to be pretty difficult. Additionally, @mwt, I suggest testing the following version, which is the one I introduced all the refactoring and subvol calculation I needed. Afterwards lots of shell-specific stuff has been changed and I wonder if the problem is either the "business logic" I changed or something related to the later commands.

https://github.com/hunleyd/btrfs-auto-snapshot/commit/60fe12034d74c00121c0ac6394211065074836b4

It might be related to my changes of course, I guess we have a slightly different BTRFS layout. How did you create yours? Looks like some default installation of some Linux distribution? I manually migrated from EXT4 at some point and simply created BTRFS manually using GPARTED and copying files over. That seems to result in a different layout regarding / vs. @, influencing what btrfs subvolume list / outputs already. So I suggest providing the following output as well:

grep btrfs /proc/mounts
grep btrfs /proc/mounts | awk '{print $2}'
btrfs subvolume list /
btrfs subvolume list EACH_LINE_OF_GREP_NR_2

This looks like the following for me:

root@potsdam:~# grep btrfs /proc/mounts
/dev/sda2 / btrfs rw,noatime,nodiratime,compress=zlib:3,ssd,space_cache,subvolid=5,subvol=/ 0 0
root@potsdam:~# grep btrfs /proc/mounts | awk '{print $2}'
/
root@potsdam:~# btrfs subvolume list /
ID 19234 gen 882901 top level 5 path .btrfs/btrfs-auto-snap_daily-2022-07-08-0800
[...]
root@potsdam:~# btrfs subvolume create /btrfs_test
[...]
root@potsdam:~# btrfs subvolume list /
ID 19234 gen 882901 top level 5 path .btrfs/btrfs-auto-snap_daily-2022-07-08-0800
[...]
ID 19495 gen 889193 top level 5 path btrfs_test

When I add some subvolume /btrfs_test, that is shown as such, pretty much like your @home. But your @ isn't shown in my list, that is different in your setup.

I as well have trouble understanding where your find-related error message comes from. Could you please try enabling tracing and print some of the output before and after the error message occurs? Attach the whole output as file if possible as well. The following should work:

# Place, Suite 330, Boston, MA  02111-1307  USA

#set -o errexit
set -o functrace
set -o xtrace

# Define our version
version=2.0.2

# Define various error codes
ERR_SUCCESS=0

[hunleyd]

./btrfs-auto-snapshot: line 468: [: : integer expression expected

That line doesn't even exist in my version, and seems to have been introduced later.

This has been fixed in the current release.

ERROR: cannot find real path for '/@': No such file or directory

This is coming from the btrfs subvolume list /@ in the local subvols line. The @ is causing the problem. If you look at btrfs subvolume list -a / you can see:

ID 258 gen 1582181 top level 5 path <FS_TREE>/@
ID 259 gen 1582181 top level 5 path <FS_TREE>/@home
ID 266 gen 1582181 top level 258 path @/srv
ID 267 gen 1147845 top level 258 path @/var/lib/portables
ID 268 gen 1147845 top level 258 path @/var/lib/machines

You can see how <FS_TREE> is being used as a placeholder for wherever you mounted it. For some reason, it doesn't seem to be interpreting that properly. I think the code needs to examine something like:

grep subvol /proc/mounts
/dev/nvme0n1p3 / btrfs rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=258,subvol=/@ 0 0
/dev/nvme0n1p3 /home btrfs rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=259,subvol=/@home 0 0

to figure out how to translate @ into a path.

@ams-tschoening when you have your usb sticks and all that inserted and mounted, what does grep subvol /proc/mounts return for you?

[hunleyd]

@mwt can i get your grep subvol /proc/mounts too?

[hunleyd]

fwiw, i'm thinking we replace btrfs subvolume list with something like awk '/subvol=/{print $4}' /proc/mounts|sed -e 's/,/\n/g'|awk -F '=' '/subvol=/{print $2}'|sed -e 's/@//'

[ams-tschoening]

This is coming from the btrfs subvolume list /@ in the local subvols line. The @ is causing the problem. If you look at btrfs subvolume list -a / you can see:

Are you sure? Aren't mountpoints forwarded to the function and in your case the MPs properly are / and /home? I would expect the source of the error message be the show instead. list returns /@, which becomes //@ because MP+subvol, /@ again because of cleanup and is forwarded to show and for that it doesn't exist as a path:

local show="$(btrfs subvolume show "${abs_path}")"

root@potsdam:~# btrfs subvolume show /not_here
ERROR: cannot find real path for '/not_here': No such file or directory

If I'm correct, that would easily explain why creating snaps works besides the error messages: Mountpoints are always added as subvols and therefore / is properly used, while /@ triggers the error. But as both are pretty much the same thing in the end, things seems to somewhat work.

keep OTOH fails, because that always works on subvols using list without given mountpoints. / vs. /@ make a difference than somewhere additionally.

What happens if you execute show with simply / and /home? Otherwise you seem correct, we need to incorporate the fact that @-subvols are mounted somewhere differently in the file system and most likely executed show with that path. That's the difference, I created my subvols in the file system view of BTRFS instead of within BTRFS' root itself.

https://askubuntu.com/a/331245/512299

@ams-tschoening when you have your usb sticks and all that inserted and mounted, what does grep subvol /proc/mounts return for you?

Have multiple different systems, but all look pretty much like the following, especially no @-paths:

root@potsdam:~# grep subvol /proc/mounts
/dev/sda2 / btrfs rw,noatime,nodiratime,compress=zlib:3,ssd,space_cache,subvolid=5,subvol=/ 0 0

We can't look at mountpoints only, that would miss subvols NOT additionally mounted and was one of the main reasons why I changed things. For example the above is the output even though I have two subvols:

• /
• /btrfs_test

root@potsdam:~# btrfs subvolume list /
ID 19234 gen 882901 top level 5 path .btrfs/btrfs-auto-snap_daily-2022-07-08-0800
[...]
ID 19495 gen 889193 top level 5 path btrfs_test

[hunleyd]

We can't look at mountpoints only, that would miss subvols NOT additionally mounted and was one of the main reasons why I changed things.

Ah, right. I knew you were using subvols but couldn't recall if they were mounted or not.

[ams-tschoening]

Does anyone know in more detail when those @-subvols are created? According my linked question it's mostly when things need to be created within the BTRFS root, like is the case for /, /home etc. In those cases the subvols will always be mounted already and found by looking at mountpoints. So @-subvols might simply be ignored in btrfs_subvols_calc, because those are handled by their mountpoints already?

That would leave us with keep, not sure what the problem is there currently.

[hunleyd]

This is coming from the btrfs subvolume list /@ in the local subvols line. The @ is causing the problem. If you look at btrfs subvolume list -a / you can see:

Are you sure? Aren't mountpoints forwarded to the function and in your case the MPs properly are / and /home? I would expect the source of the error message be the show instead. list returns /@, which becomes //@ because MP+subvol, /@ again because of cleanup and is forwarded to show and for that it doesn't exist as a path:

I think so:

╭─doug@nuc ~  ‹main✔›
╰─➤  btrfs subvolume list /@
ERROR: cannot access '/@': No such file or directory
ERROR: can't access '/@'

[hunleyd]

Does anyone know in more detail when those @-subvols are created? According my linked question it's mostly when things need to be created within the BTRFS root, like is the case for /, /home etc. In those cases the subvols will always be mounted already and found by looking at mountpoints. So @-subvols might simply be ignored in btrfs_subvols_calc, because those are handled by their mountpoints already?

That's what I'm thinking also

That would leave us with keep, not sure what the problem is there currently.

Same. Sadly, I'm about to run of of free time today and won't be able to revisit this until later in the week.

[ams-tschoening]

ERROR: cannot access '/@': No such file or directory ERROR: can't access '/@'

But that's a different error message than the original one, isn't it? :-) I get the same, but the code isn't called like that, it's called with / being the mounpoint instead of /@.

root@potsdam:~# btrfs subvolume list /not_here
ERROR: cannot access '/not_here': No such file or directory
ERROR: can't access '/not_here'

[mwt]

/dev/sdc1 / btrfs rw,relatime,compress=lzo,space_cache,subvolid=256,subvol=/@ 0 0
/dev/sdc1 /usr btrfs rw,relatime,compress=lzo,space_cache,subvolid=258,subvol=/@/usr 0 0
/dev/sdc1 /home btrfs rw,relatime,compress=lzo,space_cache,subvolid=259,subvol=/@home 0 0

[hunleyd]

But that's a different error message than the original one, isn't it? :-)

doh! yes, you're right. a perfect example of why i'm stepping awsy from keyboard for a while :)

[ams-tschoening]

/dev/sdc1 /usr btrfs rw,relatime,compress=lzo,space_cache,subvolid=258,subvol=/@/usr 0 0

Really?! :-) I would have expected that to be a subvol path of /@usr. So we have multiple cases: @XYZ being part of BTRFS root itself, like @, @home etc. I guess it's safe to assume that those are always mounted as or into the root and handled as mountpoints therefore in the calc-function.

Additionally, those @-subvols might have children on their own and those might be mounted as well, like /usr and /@/usr. But they might not as well, like /usr/foobar with a subvol path of /@/usr/foobar.

So we need to check if some mountpoint is in the list of subvols again and remove that found subvol line as it is already handled by the mountpoint itself. That might easily be possible by replacing the first / in all given mountpoints with /@ and check for each subvol line if that matches. If so, ignore that line.

That would miss /usr, as it would be expected as /@usr insead of /@/usr. So simply check in two rounds? First like above and the second with the mountpoint like /usr simply getting prefixed by /@, so that /@/usr gets checked in the end?

OTOH, we might simply check if show returns an or that special error and ignore that? :-)

Thoughts?

[mwt]

/dev/sdc1 /usr btrfs rw,relatime,compress=lzo,space_cache,subvolid=258,subvol=/@/usr 0 0

Really?! :-) I would have expected that to be a subvol path of /@usr.

Yeah. I don't know why it's this way. This is the example btrfs setup in Hetzner's documentation. I don't know if there's any advantage to /@/usr over @usr.

They probably just did it both ways (home and usr) to show that both ways are possible. Maybe I should move the usr one to the place I expected it to be.

[ams-tschoening]

Yeah. I don't know why it's this way. This is the example btrfs setup in Hetzner's documentation. I don't know if there's any advantage to /@/usr over @usr.

/@/usr is a child of /@ aka ROOT, while /@usr is at the same level like / and /home. Seems like a matter of taste mostly to me, but might be considered, as everyone is free to create such a hierarchy anyway.

[ams-tschoening]

@mwt

I don't know why it's this way. This is the example btrfs setup in Hetzner's documentation.

Could you provide the links to those docs? Didn't find them right now. Additionally, please provide the following output to further debug what might be wrong with KEEP-stuff. Don't have a test-VM yet.

btrfs subvolume list -g -o -s --sort=gen "/"
btrfs subvolume list -g -o -s --sort=gen "/" | sort -r -n -k 4 | awk '{print $NF}')
btrfs subvolume list -g -o -s --sort=gen "/" | sort -r -n -k 4 | awk '{print $NF}') | sed  -r 's!^[^/]+/.btrfs/!.btrfs/!')

btrfs subvolume list -g -o -s --sort=gen "/home"
btrfs subvolume list -g -o -s --sort=gen "/home" | sort -r -n -k 4 | awk '{print $NF}')
btrfs subvolume list -g -o -s --sort=gen "/home" | sort -r -n -k 4 | awk '{print $NF}') | sed  -r 's!^[^/]+/.btrfs/!.btrfs/!')

btrfs subvolume list -g -o -s --sort=gen "/usr"
btrfs subvolume list -g -o -s --sort=gen "/usr" | sort -r -n -k 4 | awk '{print $NF}')
btrfs subvolume list -g -o -s --sort=gen "/usr" | sort -r -n -k 4 | awk '{print $NF}') | sed  -r 's!^[^/]+/.btrfs/!.btrfs/!')

Even if error messages occur during calculating some subvols vs. mountpoints, forwarding / like you said to have done should at least work for that. And // should result in /, /home and /usr being taken into account. I guess some paths are simply unexpected, like /@ is used instead of / at some point. This makes GLOB lookups etc. simply don't find anything, so nothing will be removed ever.

But you do have some snaps created at all, don't you?

[ams-tschoening]

I came across the following interesting FSTAB, which is used to mount BTRFS subvols somewhere else as well. While no subvol names like @ are used, the underlying concept should be the same like the one breaking here. The interesting part are the subvols home_*, usr_local etc. where root subvols are mounted into some child directory and have a special name. And even this case misses something like subvol=/@/usr, with/without leading slash and stuff.

/dev/mapper/sda2_crypt /                       btrfs   defaults,noatime,autodefrag,compress=zstd,commit=120,subvol=root 0 0
UUID=xxxxxxxxxxxxxxxxx /boot                   btrfs   defaults,noatime,autodefrag,compress=lzo,commit=120 0 0
/dev/mapper/sda2_crypt /.snapshots             btrfs   defaults,noatime,autodefrag,compress=zstd,commit=120,subvol=snapshots 0 0
/dev/mapper/sda2_crypt /home                   btrfs   defaults,noatime,autodefrag,compress=zstd,commit=120,subvol=home 0 0
/dev/mapper/sda2_crypt /home/bob               btrfs   defaults,noatime,autodefrag,compress=zstd,commit=120,subvol=home_bob 0 0
/dev/mapper/sda2_crypt /opt                    btrfs   defaults,noatime,autodefrag,compress=zstd,commit=120,subvol=opt 0 0
/dev/mapper/sda2_crypt /root                   btrfs   defaults,noatime,autodefrag,compress=zstd,commit=120,subvol=home_root 0 0
/dev/mapper/sda2_crypt /srv                    btrfs   defaults,noatime,autodefrag,compress=zstd,commit=120,subvol=srv 0 0
/dev/mapper/sda2_crypt /swap                   btrfs   defaults,noatime,autodefrag,commit=120,subvol=swap 0 0
/dev/mapper/sda2_crypt /tmp                    btrfs   defaults,noatime,autodefrag,compress=zstd,commit=120,subvol=tmp 0 0
/dev/mapper/sda2_crypt /usr/local              btrfs   defaults,noatime,autodefrag,compress=zstd,commit=120,subvol=usr_local 0 0
/dev/mapper/sda2_crypt /var                    btrfs   defaults,noatime,autodefrag,compress=zstd,commit=120,subvol=var 0 0
/dev/mapper/sda2_crypt /var/lib/containers     btrfs   defaults,noatime,autodefrag,compress=zstd,commit=120,subvol=var 0 0
/dev/mapper/sda2_crypt /var/lib/libvirt/images btrfs   defaults,noatime,nodatacow,commit=120,subvol=var 0 0
/swap/swapfile         none                    swap    defaults 0 0

[hunleyd]

oh interesting @ams-tschoening ! i still don't have the bandwidth to return to this (staffing concerns at $dayjob) so are you wanting to look into this (no rush) or is it on hold for now?

[ams-tschoening]

i still don't have the bandwidth to return to this (staffing concerns at $dayjob)[...]

Same problem here, but at least have it on my TODO list. So let's keep it for me for now and if you don't hear back, but think to have time to look into this on your own, it most likely make sense to do so.

[ams-tschoening]

@mwt I've provided a PR which should(!) cover your use-cases now and should(! ;-)) be wort some tests:

https://github.com/hunleyd/btrfs-auto-snapshot/pull/9

[hunleyd]

pr #9 looks good to me, but i'll hold approval until @mwt can confirm it fixes the reported issue for them