Support for fanotify

[necouchman]

It appears, based on a couple of web pages and some testing, that ZFSOnLinux does not currently support the newer fanotify for file access/change notifications. Several backup and malware scanning products are able to make use of this functionality in Linux to provide either file-level backups or on-access malware scanning capability with lower performance penalties. It would be a nice addition to ZFSonLinux.

[tuxoko]

What kind of test makes you think it's not supported? The notify stuff should be handled in the VFS layer, so you should be able to use it on ZFS.

[necouchman]

I have a couple of sources for this:

• Testing ClamAV's OnAccess scanning support with ZFS. I have a CentOS 7 system set up, the kernel supports fanotify, and I can successfully detect the EICAR test virus on an ext4 filesystem by doing a "cat eicar.com" and get an error in the clamd log file indicating that a virus was located. On ZFS on exactly the same host, I get the following error during clamd startup:

ERROR: ScanOnAccess: Could not watch path '/fstest', Success

And, if I try to access a file on one of the ZFS file systems, I get no warnings in the log files. I've tried with a variety of atime/relatime options to see if that impacts the detection, and it does not.

• Sophos has a support page that lists the status of their On-Access A/V with various Linux filesystems. It has two columns for each filesystem - one for Talpa Support, which I believe is their own module they inject into the kernel to do on-access detection, and one for Fanotify support. Fanotify is listed as a "yes" for several filesystems, including btrfs, ext2/3/4, jfs, xfs, fuse, but listed as a no for, among other things, zfs. This page does not specify what variant of zfs they're talking about, so it could be they're referring to an implementation of ZFS other than ZFSonLinux, but I believe most of the other ones are FUSE-based, so presumably they would fall into the FUSE category, instead.

[tuxoko]

@necouchman Then, you need to contact the AV developer to see what's the problem. Because there's no reason fanotify wouldn't work on ZFS.

[haasn]

Running fatrace from the command line makes it pretty obvious that fanotify + zfs does not work as expected. I can do I/O in zfs all day long and I get no events from fatrace whatsoever. But any I/O to e.g. an ext4, xfs or vfat partition works fine.

Try it yourself if you don't believe me: install fatrace and run it. Then touch or stat or ls any file on zfs and you should get output if it was working.

[haasn]

Okay, I think the issue is that fatrace is ignoring the zfs mount type by default for some reason. Using fatrace -c gets it to work. Probably they have a whitelist somewhere of filesystems to monitor.

[RubenKelevra]

Okay, I think the issue is that fatrace is ignoring the zfs mount type by default for some reason. Using fatrace -c gets it to work. Probably they have a whitelist somewhere of filesystems to monitor.

Seems to be broken now :(

fatrace -c
fatrace: Received unexpected event info type 0

[nh2]

I can confirm that both fatrace and fsnoop do not work on a ZFS 0.8.3 root file system.

[necouchman]

Re-opening, since it seems like I'm not the only one experiencing this issue, and it isn't just with A/V.

[lastquestion]

Hi, I found this issue via google because I was trying to see what files smbd was accessing on my ZFS pools and I was, like everyone else, super surprised that fatrace didn't show activity.

fatrace does work when called with -c, but your current working directory must be inside the zfs mount point for the zpool you expect io in. For example, you cannot be in /tank if there is a nested pool /tank/magic where file access are occurring; you must be in /tank/magic and then run fatrace -c.

Unfortunately, the fatrace installed via apt-get in ubuntu focal doesn't have a -v option. So, just so others can see what version this works on, I have fatrace@0.13-2 according to dpkg.

I believe that @haasn's conjecture is correct and fatrace is ignoring zfs mount types. Assuming the code base mirrored on github matches 0.13 - https://github.com/truncs/fatrace/blob/master/fatrace.c - which is of course hard to tell... but at least the name on the repo matches the AUTHOR in the debian package, so that's good...

Well, anyway, clearly the line that is problematic is here https://github.com/truncs/fatrace/blob/master/fatrace.c#L180

the expected behavior here was that the filesystem name is an actually accessable path in /dev/ somewhere, which is true for most real filesystems, but for ZFS mounts this clearly not true. This is actually a non-trivial problem to solve, because if you try to check on filesystem types, well, there's infinitely many, so you'd have to have either a list you'll allow or reject on.

You can try my fork, where I simply just remove that check. Works fine to see accesses on all ZFS mounts. As a side effect it will show you accesses to things like proc and other non-real-fs.

https://github.com/lastquestion/fatrace

Ultimately I think the fix needs to be in fatrace; probably the cleanest thing to do is to drop that check entirely, and maybe add an option to filter on specific filesystem types. Perhaps even encode a list of usual proc/mem/virtual FS types as a default... but that would be up to Martin Pitt.

[mocukie]

ZFS does not seem to support the new fanotify features in linux kernel 5.1. The example fanotify_fid.c does not work, fanotify_event_info_fid is not set correctly.

Test on ubuntu 20.04 with ZFS 0.8.3.

[devZer0]

yes, please fix this, i'm using fanotify a lot and after last zfs update, it broke.

[MatthiasKuehneEllerhold]

Im testing fanotify with ZFS in Debian 11 Bullseye ... it doesnt seem to work. Neither 2.0.3 nor 2.1.11 works! Please fix it if you can.

[glibg10b]

sudo fatrace | grep /media/pool/ works correctly on my system with the default pool feature flags.

$ zfs version
zfs-2.2.0-0ubuntu1~23.10.3
zfs-kmod-2.2.0-0ubuntu1~23.10.2

[MatthiasKuehneEllerhold]

I can confirm this works now: Debian 12 bookworm and

zfs-2.2.4-1~bpo12+1
zfs-kmod-2.2.3-1~bpo12+1

Thanks!!

[devZer0]

i can also confirm fatrace 0.17.0-1 works with zfs-kmod-2.2.4-1~bpo12+1 again on debian 12.5