sys_getdents64 fails when using appname | pid flag

[glitsj16]

Hi, I installed the AUR package and wanted to play wih this interesting tool on my (fully updated) Arch Linux box. Sadly I'm seeing the below errors and wanted to ask if I need to change something to get it working as expected.

I took out the color codes from the output for readibility here. Seeing the same output when using appname or PID flag. Adding -vv doesn't give more details, neither does a debug build, so I'm a bit at a loss as to what might be going wrong. Any advice would be very welcome!

$ gcc --version
gcc (GCC) 12.2.1 20230111
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ inotify-info geany
------------------------------------------------------------------------------
INotify Limits:
  max_queued_events    16384
  max_user_instances   1024
  max_user_watches     100000
------------------------------------------------------------------------------
        Pid  App                        Watches   Instances
        1067 systemd                         52   3
        1341 gsd-xsettings                   23   1
        1757 thunar                          21   1
        1222 gnome-session-binary             6   2
        1743 gnome-pie                        5   1
        1756 python2.7                        4   2
        1089 dbus-daemon                      3   1
        3678 journalctl                       3   1
        1509 pulseaudio                       2   2
        6632 tumblerd                         1   1
        6389 xfce4-terminal                   1   1
        4775 geany                            1   1
        3755 inotifywait                      1   1
        3735 inotifywait                      1   1
        1815 inotifywait                      1   1
        1809 inotifywait                      1   1
        1444 devilspie2                       1   1
        1333 gsd-color                        1   1
        1245 gvfs-udisks2-volume-monitor      1   1
        1076 nss-tlsd                         1   1
------------------------------------------------------------------------------
Total inotify Watches:   130
Total inotify Instances: 25
------------------------------------------------------------------------------

Searching '/' for listed inodes... (2 threads)
ERROR: sys_getdents64 failed on '/proc/555/net/': -1 errno:22
ERROR: sys_getdents64 failed on '/proc/1227/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/3031/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/3127/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/3353/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/3407/map_files/': -1 errno:13
[hangs here until Ctrl-C]

[mikesart]

Interesting. Guess we could skip the /proc filesystem? Wonder why this isn't hanging and/or spewing these warnings on my Debian Testing system. Could you run it under gdb and hit Ctrl-C and see where it's hanging if you get a chance? Thanks!

[glitsj16]

Thanks for your response. Here's wha gdb comes up with:

$ gdb inotify-info
[...]
(gdb) run geany
[...]
Searching '/' for listed inodes... (2 threads)
[New Thread 0x7ffff7a186c0 (LWP 52660)]
ERROR: sys_getdents64 failed on '/proc/555/net/': -1 errno:22
ERROR: sys_getdents64 failed on '/proc/1227/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/3031/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/3127/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/3353/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/3407/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/47710/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/47773/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/47795/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/48164/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/48199/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/48200/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/48301/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/50509/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/50542/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/51402/map_files/': -1 errno:13
ERROR: sys_getdents64 failed on '/proc/51438/map_files/': -1 errno:13
^C
Thread 1 "inotify-info" received signal SIGINT, Interrupt.
0x00007ffff7b19abd in syscall () from /usr/lib/libc.so.6
(gdb)

Arch Linux has glibc 2.36 by the way:

$ pacman -Q glibc
glibc 2.36-7

[mikesart]

Sorry, could you do a list and bt in gdb? Curious what the stack trace looks like.

Talked with @scottlu and we're thinking that we may ignore the /proc filesystem by default and add a command line option to force scanning that as well. Would love to hear thoughts from anyone on this though. Thanks glitsj16.

[glitsj16]

Here's the requested list and bt:

^C
Thread 1 "inotify-info" received signal SIGINT, Interrupt.
0x00007ffff6b100a0 in __libc_open64 (file=0x6040000d7250 "/mnt/crow16/lib/systemd/system-sleep/", oflag=65536) at ../sysdeps/unix/sysv/linux/open64.c:41
41    return SYSCALL_CANCEL (openat, AT_FDCWD, file, oflag | O_LARGEFILE,
(gdb) list
36        va_start (arg, oflag);
37        mode = va_arg (arg, int);
38        va_end (arg);
39      }
40
41    return SYSCALL_CANCEL (openat, AT_FDCWD, file, oflag | O_LARGEFILE,
42               mode);
43  }
44
45  strong_alias (__libc_open64, __open64)
(gdb) bt
#0  0x00007ffff6b100a0 in __libc_open64 (
    file=0x6040000d7250 "/mnt/crow16/lib/systemd/system-sleep/", oflag=65536)
    at ../sysdeps/unix/sysv/linux/open64.c:41
#1  0x00005555556906d9 in thread_info_t::parse_dirqueue_entry() () at inotify-info.cpp:406
#2  0x000055555569136e in parse_dirqueue_threadproc(void*) () at inotify-info.cpp:483
#3  0x0000555555695067 in find_files_in_inode_set(std::__debug::vector<procinfo_t, std::allocator<procinfo_t> > const&, std::__debug::vector<filename_info_t, std::allocator<filename_info_t> >&) () at inotify-info.cpp:656
#4  0x0000555555697b18 in main () at inotify-info.cpp:831

Notice that /mnt/crow16 is a SSHFS mount, I have a few of those. After stopping their systemd mount services I also double-checked a few sysctl knobs that are enabled by default on this box. And sure enough, one of those seems to be the culprit: kernel.randomize_va_space. It's set to 1 but when set to 0 (disabled) inotify-info starts to do its thing and happily reports where my geany editor is keeping a watch on:

525216 [8:4] /home/glitsj16/.local/share/

Hopefully this info is worth something to the devs. Feel free to ask if I can offer anything else.

Cheers!

[mikesart]

That's great info - I'll try to look at this next week when I get back from our company meeting. Thanks again!

[glitsj16]

Talked with @scottlu and we're thinking that we may ignore the /proc filesystem by default and add a command line option to force scanning that as well.

Sorry to bother, but is there any progress on ignoring /proc yet? Or a WIP branch on that work? The /proc scanning (and consequent errors) is actually the only thing breaking inotify-info on Arch Linux at the moment.

[mikesart]

Well, not sure exactly how to handle this. It sounds like the sshfs mount is causing the issue for you and not /proc?

So I assume this means we need to look at the mount table, parse that and then ignore sshfs mount points?

[glitsj16]

Ah that is indeed correct. Grepping for fuse.sshfs or just sshfs in mount output does provide a way to check if a SSHFS mountpoint exists. My confusion stems from not knowing why scanning the whole /proc is actually needed for correct functioning of inotify-info. I'm assuming it is not. If that assumption is correct there's a nice speed gain possible by excluding it entirely from scanning. Potentially others like /boot, /dev, /media, /mnt, /run, /sys, /tmp and /var might be candidates for skipping too?

[mikesart]

Can you try the latest (commit 8add69e) when you get a chance? Should skip /proc and hopefully fuse dirs.

[glitsj16]

Can you try the latest (commit 8add69e) when you get a chance? Should skip /proc and hopefully fuse dirs.

Confirming it indeed skips /proc and fuse mountpoints now. Smooth as butter! Thank you very much for your time and effort in dealing with this. I'll close the issue.

[mikesart]

Great to hear & thanks for the detailed info!