Shows no file activity when reading/writing to HDD from Shell

[porg]

Reproduction

In one Terminal window I am already on my NAS via SSH as root. There I have a fatrace up for monitoring: (with sed shortening the ultra long disk-uid for better readability)

$ fatrace | sed -En 's!/dev-disk-by-uuid-[^/]+/!/•••/!p' | uniq
  < Prompt open. No output follows. Currently no file access on the HDD. >

In another Terminal window I now connect to my NAS via SSH as me (myself)

$ ssh me@odroid.local 
me@odroid.local's password: 

Welcome Screen
Last login: <date/time> from <IP-address>

I now ls -l the HDD filepath:

me@odroid:~$ l /srv/dev-disk-by-uuid-LONG-UID/

👉 No file events in fatrace.

I now press TAB for autocompletion: (which also means a read operation)

aquota.group       aquota.user        Compose/           dockerhome/        lost+found/        Plex/              Shared/

👉 No file events in fatrace.

I now ls -l a bit deeper into the HDD filepath:

me@odroid:~$ l /srv/dev-disk-by-uuid-LONG-UID/Plex/
custom-cont-init.d/ custom-services.d/     Library/             tmp-transcoding/    

👉 No file events in fatrace.

Now let's do some writing in the shell to the HDD filepath

me@odroid:~$ echo "1" >> /srv/dev-disk-by-uuid-LONG-UID/test.txt
me@odroid:~$ echo "2" >> /srv/dev-disk-by-uuid-LONG-UID/test.txt
me@odroid:~$ echo "3" >> /srv/dev-disk-by-uuid-LONG-UID/test.txt
me@odroid:~$ cat /srv/dev-disk-by-uuid-LONG-UID/test.txt
1
2
3

👉 No file events in fatrace.

Why does fatrace show no file activity when this comes from a Shell?

• All other file activities (from Samba, Plex, Docker, …) it shows just fine.

Environment

• Open Media Vault 6 (based on Debian 11)
  • On an Odroid Home Cloud 2 (SBC: 32-bit CPU + SD card + SATA HDD)
  • Using https://www.armbian.com/odroid-xu4/
  • Armbian 22.05.3 Bullseye with Linux 5.4.199-odroidxu4
• fatrace 0.16.2-2 (installed via apt)

[martinpitt]

Some thoughts:

• Do you get events without the sed | uniq pipeline, i.e. when fatrace writes directly to the TTY? This uses line buffering, while pipelining switches to much larger buffers (usually around 1K or so). Version 0.16.3 improved this so that pipelining/sed etc. works as expected, but you are still using the previous version.
• If that's not it, what's the file system on that /srv/dev-disk-by-uuid-LONG-UID mountpoint? If it's zfs or btrfs, these have known problems due to kernel limitations.
• If that's not it either, can you please build fatrace with changing #define DEBUG 1 here, and give me the output?

Thanks!

[porg]

Sorry that I forgot to mention: $ fatrace | sed -En 's!/dev-disk-by-uuid-[^/]+/!/•••/!p' | uniq

• /srv/dev-disk-by-uuid-LONG-UID is an EXT4
• And the sed filtering I used all the time and worked fine all the time, e.g.
  • SMB access to Shares on it
  • Plex server scanning/serving files form there
• Just when performing write operations via shell (e.g. echo "bla" > /srv/dev-disk-by-uuid-LONG-UID/file.txt I strangely get no output.
• Just line 48 needs to be uncommented, or the whole block until line 53 ?

[martinpitt]

I meant if you just call fatrace, do you get events for that device?

Otherwise, try the ARM binary from https://github.com/martinpitt/fatrace/issues/19#issuecomment-1186807249 , that includes the buffering improvement.

For enabling debugging, I meant that you replace the existing line 48

#define DEBUG 0

with

#define DEBUG 1

and build.

[porg]

Seems fine

• I now ran the self built fatrace from (git clone ca 30min ago, must have been commit af3ebbd5bed67fc490f7aca6d7e2f78e33851759)
• ✅ My file write events from the shell both as normal admin user and root were properly reflected in fatrace.
• ℹ️ File writes to tmpfs from both users remained unregistered. This is normal and by design I assume.
• ❓ File deletion events triggered by "rm test-*" were not picked up by fatrace with the grep "test- pipe after it at least, which I had to use b/c currently there was too much noise on my system going on.
  • Is it normal that they do not get picked up?
  • If not, please give me a sample line of how such an event should look line, then I may understand why it must have slipped through my grep filter.

Log of my tests

## As ME in one Terminal window I ran: $ sudo ./fatrace | grep 'test-' ## In another Terminal window as ME I than ran: $ echo "test by me" >> ~/test-rootfs-homedir.txt $ echo "test by me" >> /srv/dev-disk-by-uuid-LOND-UID/Shared/test-hdd-me.txt $ echo "test by me" >> /tmp/test-tmpfs-me.txt ## In another Terminal window as ROOT I ran: echo "test by root" >> ~/test-rootfs-homedir.txt echo "test by root" >> /srv/dev-disk-by-uuid-LOND-UID/Shared/test-hdd-root.txt echo "test by root" >> /tmp/test-tmpfs-root.txt ## The terminal window running fatrace showed all my write activities nicely ### The write operations of ME bash(22309): CWO /srv/dev-disk-by-uuid-LOND-UID/Shared/testhdd.txt bash(22309): CWO /srv/dev-disk-by-uuid-LOND-UID/Shared/testhdd.txt bash(22309): O /srv/dev-disk-by-uuid-LOND-UID/Shared/testhdd.txt bash(22309): CW /srv/dev-disk-by-uuid-LOND-UID/Shared/testhdd.txt bash(22309): O /srv/dev-disk-by-uuid-LOND-UID/Shared/test-hdd.txt bash(22309): CW /srv/dev-disk-by-uuid-LOND-UID/Shared/test-hdd.txt bash(22309): CWO /srv/dev-disk-by-uuid-LOND-UID/Shared/test-hdd.txt bash(22309): CWO /home/sn/test-rootfs-homedir.txt bash(22309): CWO /home/sn/test-rootfs-homedir.txt ### The write operations of ROOT bash(23827): O /root/test-rootfs-homedir.txt bash(23827): CW /root/test-rootfs-homedir.txt bash(23827): O /root/test-rootfs-homedir.txt bash(23827): CW /root/test-rootfs-homedir.txt bash(23827): CWO /root/test-rootfs-homedir.txt bash(23827): O /root/test-rootfs-homedir.txt bash(23827): CW /root/test-rootfs-homedir.txt bash(23827): CWO /srv/dev-disk-by-uuid-LOND-UID/Shared/test-hdd-root.txt bash(23827): CWO /srv/dev-disk-by-uuid-LOND-UID/Shared/test-hdd-root.txt bash(23827): CWO /srv/dev-disk-by-uuid-LOND-UID/Shared/test-hdd-root.txt

[martinpitt]

File deletion events triggered by "rm test-*" were not picked up

You will get an event on the directory that contained the file you removed, not the file itself. Sorry, that's all that Linux gives -- as the thing that is modified with a deletion is really the containing directory. The file itself is never touched.

[porg]

Ok. Good to know. If needed to test for file deletions one day one could then find some workaround, e.g. to check for the non-existence of a file and then see whether its parent directory changed its change-timestamps, and from this can deduce that a deletion must have taken place.

This addressed my last open question. The rest worked fine.

→ Closing issue. Thanks for the conversation!