fantoify fsid mark failing for btrfs subvolumes: Invalid cross-device link

[ohsix]

probably already know this from the failing test but,

fanotify_test_fid returns -EXDEV on btrfs

https://github.com/torvalds/linux/blob/7d6beb71da3cc033649d641e1e608713b8220290/fs/notify/fanotify/fanotify_user.c#L1075

the older version without FAN_REPORT_FID was working until I upgraded the os

[root@pro ohsix]# fatrace 
fatrace: Failed to add watch for /: Invalid cross-device link
DEBUG: mount / fd 4
DEBUG: ignore: fsname: sysfs dir: /sys type: sysfs
DEBUG: ignore: fsname: proc dir: /proc type: proc
DEBUG: ignore: fsname: devtmpfs dir: /dev type: devtmpfs
DEBUG: ignore: fsname: securityfs dir: /sys/kernel/security type: securityfs
DEBUG: ignore: fsname: tmpfs dir: /dev/shm type: tmpfs
DEBUG: ignore: fsname: devpts dir: /dev/pts type: devpts
DEBUG: ignore: fsname: tmpfs dir: /run type: tmpfs
DEBUG: ignore: fsname: cgroup2 dir: /sys/fs/cgroup type: cgroup2
DEBUG: ignore: fsname: pstore dir: /sys/fs/pstore type: pstore
DEBUG: ignore: fsname: none dir: /sys/fs/bpf type: bpf
DEBUG: ignore: fsname: none dir: /sys/kernel/tracing type: tracefs
DEBUG: ignore: fsname: selinuxfs dir: /sys/fs/selinux type: selinuxfs
DEBUG: ignore: fsname: systemd-1 dir: /proc/sys/fs/binfmt_misc type: autofs
DEBUG: ignore: fsname: debugfs dir: /sys/kernel/debug type: debugfs
DEBUG: ignore: fsname: mqueue dir: /dev/mqueue type: mqueue
DEBUG: ignore: fsname: hugetlbfs dir: /dev/hugepages type: hugetlbfs
DEBUG: ignore: fsname: fusectl dir: /sys/fs/fuse/connections type: fusectl
DEBUG: ignore: fsname: binfmt_misc dir: /proc/sys/fs/binfmt_misc type: binfmt_misc
DEBUG: ignore: fsname: configfs dir: /sys/kernel/config type: configfs
DEBUG: ignore: fsname: tmpfs dir: /tmp type: tmpfs
DEBUG: add watch for btrfs mount /home
fatrace: Failed to add watch for /home: Invalid cross-device link
DEBUG: mount /home fd 6

[martinpitt]

probably already know this from the failing test but,

No, which one is that? The CI here runs on GitHub workflows, which is Ubuntu 20.04 on ext4 (... I guess), and it is happy.

So that means that fanotify_init (FAN_CLASS_NOTIF | FAN_REPORT_FID, O_LARGEFILE) succeeds (it should give EINVAL if it's not supported), but fanotify_mark() fails then?

I added a possible fix to https://github.com/martinpitt/fatrace/tree/btrfs , want to try that?

I won't land it yet, I want to try and reproduce this in a test (on a loop device).

[martinpitt]

I just added a test for btrfs, on a loop device. This works fine, and also that's not too surprising -- after alll, there is nothing cross-device there. I figure you have something slightly more fancy? Subvolumes? /home being a symlink or bind-mount or something?

[ohsix]

doh ya.

/ and /home are two subvolumes on the same device /dev/sda2 on / type btrfs (rw,relatime,seclabel,ssd,space_cache,subvolid=100027,subvol=/root) /dev/sda2 on /home type btrfs (rw,relatime,seclabel,ssd,space_cache,subvolid=100026,subvol=/home)

the failing test was CI, one of the containers. called "btrfs tests" now (don't remember if it was at the time)

as for the btrfs branch: it does the same thing fatrace did before the patch, runs but there's no events from / or /home

the thing that made it confusing is that fatrace runs and doesn't exit, it just didn't print any events for the expected filesystems (when run with no options) it sees and sets up monitors on squashfs files snap has mounted, but the events on those filesystems are relatively rare so it wasn't easy to see that was the case

don't know much about fanotify but if there are filesystems or mounting scenarios where a combination of flags won't work it would be useful to have a way for fatrace to say so, like printing a table of all the filesystem watches and the actual flags being used. and maybe a strict/loose option to make fatrace exit when the flags you ask for can't be done on the filesystem(s)

that leaves the door open for running fatrace with no options and showing some events on btrfs and more where it is able (which is 99% of my use-case, just to take a peek at what's going on)

thank you for looking into this

[ohsix]

sorry for the delay,

here's everything it says in debug mode before pausing for lack of filesystem activity:

[ohsix@pro fatrace]$ sudo ./fatrace DEBUG: FAN_MARK_FILESYSTEM not supported; falling back to FAN_MARK_MOUNT fatrace: Failed to add watch for /: Invalid argument DEBUG: mount / fd 4 DEBUG: ignore: fsname: sysfs dir: /sys type: sysfs DEBUG: ignore: fsname: proc dir: /proc type: proc DEBUG: ignore: fsname: devtmpfs dir: /dev type: devtmpfs DEBUG: ignore: fsname: securityfs dir: /sys/kernel/security type: securityfs DEBUG: ignore: fsname: tmpfs dir: /dev/shm type: tmpfs DEBUG: ignore: fsname: devpts dir: /dev/pts type: devpts DEBUG: ignore: fsname: tmpfs dir: /run type: tmpfs DEBUG: ignore: fsname: cgroup2 dir: /sys/fs/cgroup type: cgroup2 DEBUG: ignore: fsname: pstore dir: /sys/fs/pstore type: pstore DEBUG: ignore: fsname: none dir: /sys/fs/bpf type: bpf DEBUG: ignore: fsname: none dir: /sys/kernel/tracing type: tracefs DEBUG: ignore: fsname: selinuxfs dir: /sys/fs/selinux type: selinuxfs DEBUG: ignore: fsname: systemd-1 dir: /proc/sys/fs/binfmt_misc type: autofs DEBUG: ignore: fsname: mqueue dir: /dev/mqueue type: mqueue DEBUG: ignore: fsname: hugetlbfs dir: /dev/hugepages type: hugetlbfs DEBUG: ignore: fsname: debugfs dir: /sys/kernel/debug type: debugfs DEBUG: ignore: fsname: fusectl dir: /sys/fs/fuse/connections type: fusectl DEBUG: ignore: fsname: configfs dir: /sys/kernel/config type: configfs DEBUG: ignore: fsname: binfmt_misc dir: /proc/sys/fs/binfmt_misc type: binfmt_misc DEBUG: ignore: fsname: tmpfs dir: /tmp type: tmpfs DEBUG: add watch for btrfs mount /home fatrace: Failed to add watch for /home: Invalid argument DEBUG: mount /home fd 6 DEBUG: add watch for ext4 mount /boot fatrace: Failed to add watch for /boot: Invalid argument DEBUG: mount /boot fd 7 DEBUG: ignore: fsname: tmpfs dir: /run/user/1000 type: tmpfs DEBUG: ignore: fsname: gvfsd-fuse dir: /run/user/1000/gvfs type: fuse.gvfsd-fuse DEBUG: ignore: fsname: tracefs dir: /sys/kernel/debug/tracing type: tracefs DEBUG: ignore: fsname: portal dir: /run/user/1000/doc type: fuse.portal

[ohsix]

any news on this? if it helps the expected behavior of -c is also busted :)

[ohsix@pro /]⛈️ fatrace -c
fatrace: Failed to add watch for .: Invalid cross-device link

this is a default fedora install with the default volume manager / btrfs options, gnome-boxes can download the iso and install it for you pretty quick. or if there's anything i can collect here please let me know

[martinpitt]

Sorry for the delay! I see this as well now on a Fedora 35 cloud image, which uses btrfs with subvolumes.

This is also easy to reproduce with the integration test:

--- a/tests/fatrace-btrfs
+++ b/tests/fatrace-btrfs
@@ -16,7 +16,9 @@ mkdir -p "$MOUNT"
 mount -o loop "$IMAGE" "$MOUNT"
 trap "umount -l '$MOUNT'" EXIT INT QUIT PIPE

-cd "$MOUNT"
+btrfs subvolume create "$MOUNT/subv1"
+
+cd "$MOUNT/subv1"

 echo "hello" > world.txt

However, I'm afraid I wouldn't know what to do here.. both the new and old fanotify_mark() APIs return EXDEV/EINVAL, so I am at loss how else to watch events on btrfs subvolumes. Looks like the fanotify API just simply does not support this?

Note that the "pausing for lack of fs activity" only happens if you run it without options -- as long as there is at least one watched fs, it will wait for that. With e.g. --current-mount, it will exit immediately with the error.

[martinpitt]

Ah, it's not actually that bad.. I can runfatrace --current-mount on a mount of the btrfs root, just not in any mounted subvolume. That watch will still get the subvolume events, I added a test in commit 061b41469d2c904997a23e545c50cba19241967d.

Now, I realize that does not help much if you only have subvolumes mounted, and not the root device -- but it seems this is the best that Linux can do.

[ohsix]

i think I can adapt to that, thank you for looking into it

i looked at the man pages again and they do remark about EXDEV/NODEV & btrfs, here's where it was added https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/man2/fanotify_mark.2?id=0a4db6dc742d9150d73048e889de9e6accc53d46

can see that it might be a semantic change but i don't know any of this well enough to say that it could work like it did before

[lukefor]

It seems to be a popular choice for distros to use subvolumes by default now on btrfs, even Debian's default of putting everything in an @ subvolume triggers this bug

A quick and dirty workaround is to mount the root of the root partition (i.e. mount without subvol= parameter) elsewhere before running fatrace. As confirmed above this correctly picks up writes to subvolumes

[chapmanjacobd]

A quick and dirty workaround is to mount the root of the root partition (i.e. mount without subvol= parameter) elsewhere before running fatrace

I don't often mount things by UUID; leaving this here in case it is helpful for others:

sudo mount -t btrfs UUID=69d63d6b-4293-4c20-bd0e-3595fab38616 /mnt/fatrace/

[WildPenquin]

Hi,

I'm a bit lost here, and I can not use fatrace at all on btrfs.

All my mounts are in subvolumes, the root subvolume (ID=5) is empty and not mounted anywhere. The default subvolume (which gets mounted without any subvol= -parameter) has been changed. It is advised in many places to not use the default top-level subvolume (ID 5) since it canmake managing subvolumes a bit more challenging.

However, from the above I underestod I need to mount the default subvolume somewhere else, but perhaps what was meant to say, is I need to mount the top-level subvolume?

Either way, after running sudo mount -t btrfs UUID="6bf58237-08c3-4b44-a637-85c71ae14a17" /mnt/fatrace, I get:

$ fatrace 
fatrace: Failed to add watch for /: Invalid cross-device link
fatrace: Failed to add watch for /home: Invalid cross-device link
fatrace: Failed to add watch for /mnt/fatrace: Invalid cross-device link

And fatrace can not producing any output at all.

sudo btrfs subvolume get-default /             
ID 272 gen 223303 top level 5 path @

This is on 0.17.0-1 installed from AUR.

I should probably add that the subvolume layout is quite complex; every user has own subvolumes in /home, and many subvolumes have automated snapshots (subvolumes in .subvolume subvolume).

[chapmanjacobd]

Yes, as far as I understand it needs to be the top-level subvolume:

sudo mount UUID=$(lsblk -no UUID $(df -P / | awk 'END{print $1}')) /mnt/fatrace/

Does mount grab options from /etc/fstab in the above command? No. The man page says this:

The mount program does not read the /etc/fstab file if both device (or LABEL, UUID, ID, PARTUUID or PARTLABEL) and dir are specified.

This is what it looks like for me, after mounting with the above command:

findmnt /mnt/fatrace/
TARGET       SOURCE                                                FSTYPE OPTIONS
/mnt/fatrace /dev/mapper/luks-ad426a7b-d653-4ef7-b803-ce2712c739d8 btrfs  rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/

So it looks like it is using the / subvol. (I guess those additional options are being loaded from device metadata outside of the user filesystem rather than from /etc/fstab? None of the entries in my fstab have subvol=/)

I don't think this is the same as the root subvol because I can see root inside of the mountpoint:

ls /mnt/fatrace/
Permissions Size User Date Modified Name
drwxr-xr-x@    - root 26 Apr 09:18  home/
dr-xr-xr-x@    - root 26 Apr 14:36  root/

hope this helps!

Also, make sure you do sudo fatrace otherwise you'll get: fatrace: Failed to add watch for /mnt/fatrace: Operation not permitted

[WildPenquin]

Thanks, I will try wit the root volume later.

I'm not sure how fstab is relevant here (or were you replying to someone else?). I've se the default sobvolume with btrfs subvolume set-default, which I though is common practice. Earlier it was mentioned:

A quick and dirty workaround is to mount the root of the root partition (i.e. mount without subvol= parameter)

which is a bit confusing, but as "without subvol" can be really anything, so that's not what is actually meant here.

EDIT: Indeed what needs to be done is to mount the root, not the default. It's quite obvious if one reads carefully the earlier discussion, the only cause of confusion is the quote above, as indeed the root might not be the default.