Excclusion / Inclusion filters do not seem to work on Linux

[AngmarLord]

fswatch version - 1.14.0

Environment

CentOs 8.0 GCC version - 8.3.1

Description

The exclude, include filters do not seem to work. Assuming a file extension of the type .abc, exclusion of all, followed by inclusion of .abc, as indicated in documentation, does not create any events at all.

I tried several formats -

fswatch -x --event Created -e ".*" -i "\\.ext$" /path

and

fswatch -x --event Created -e '.*\.*$' -i '.*\.abc$' /path

apart from other variations. In most cases, I do not receive any events, using -v, it seems using an exclusion filter like those described above stops fswatch from adding the /path to watch-list.

Gets stuck at - start_monitor: Adding path: /home/bob1

Do not get the message - add_watch: Added: /home/bob1

[jleider]

This is happening to me as well on Ubuntu 20.04 server, fswatch version 1.14.0.

As soon as I add the -e ".*" nothing will be picked up by fswatch

fswatch -1 -e ".*" -i "\.ext\$" /path works in mac but not on linux.

[mcarpenter]

Also here. I have a Makefile using fswatch. Last successfully used some time last year. I subsequently upgraded to ubuntu 20 and it no longer works, with symptoms as above.

Ubuntu 20 has fswatch 1.14.0. Launchpad indicates that ubuntu 18 had 1.11.2.

I rebuilt 1.11.3 (last release of minor version 1.11) and installed into /opt, and that works fine.

Conclusion: either a regression in fswatch >1.11.3 or a problem in the Ubuntu 20 packaging.

[xxxserxxx]

Conclusion: either a regression in fswatch >1.11.3 or a problem in the Ubuntu 20 packaging.

It's a regression, because it's also happening on Arch (fswatch 1.16.0-1).

[mcarpenter]

Bisection indicates commit b04d0edc1dfb21e08beeb594378176e1812e997a "Do not scan a directory tree if the directory name is excluded by filters"

This appears to affect inotify (Linux) but also kqueue and poll monitors. The change for all 3 is equivalent, in the recursive scan() function:

commit b04d0edc1dfb21e08beeb594378176e1812e997a (HEAD, refs/bisect/bad)
Author: Enrico M. Crisostomo <enrico.m.crisostomo@gmail.com>
Date:   Mon May 7 00:33:58 2018 +0200

    Do not scan a directory tree if the directory name is excluded by filters

diff --git a/libfswatch/src/libfswatch/c++/inotify_monitor.cpp b/libfswatch/src/libfswatch/c++/inotify_monitor.cpp
index 7b60092..aee31a2 100644
--- a/libfswatch/src/libfswatch/c++/inotify_monitor.cpp
+++ b/libfswatch/src/libfswatch/c++/inotify_monitor.cpp
@@ -165,7 +165,7 @@ namespace fsw
      */
     if (!is_dir && !accept_non_dirs) return;
     if (!is_dir && directory_only) return;
-    if (!is_dir && !accept_path(path)) return;
+    if (!accept_path(path)) return;
     if (!add_watch(path, fd_stat)) return;
     if (!recursive || !is_dir) return;
[snip]

Excluding everything (-e '.*') then excludes the top level directory (path), since this (changed) line returns before the watch can be added (next line).

I don't see a generic workaround at the user level.

Perhaps scan() should special-case the initial, top-level call? There is already a parameter accept_non_dirs that is true on the initial call to "to allow watching a file when first invoked on a node". Perhaps this could be renamed (eg top_level_node) and used for both purposes?

A possible downside to that is that if a user passes a directory on the command line that matches an exclude flag then it... won't be excluded. That might be surprising.

[xxxserxxx]

@mcarpenter Because you specifically talk about exclusion filters, I want to be clear that inclusion filters also don't work; this isn't an issue only if an exclusion is specified. E.g., if all you want to watch for are .ics files, the following does not work:

$ mkdir fswatchtest
$ (sleep 5 ; touch fswatchtest/bar) &
$ fswatch --include '.*\.ics$'  --event Created --event IsFile fswatchtest
/home/user/fswatchtest/bar

The documentation is not clear on this point, but I am assuming that -i and -I imply excluding everything that doesn't match the include pattern. If -i needs -e to have effect, then the code @mcarpenter found prevents -i from working, and it'd be good to have that documented as it would be an uncommon behavior for POSIX tools that provide include/exclude functionality.

[mcarpenter]

@xxxserxxx wrote:

I want to be clear that inclusion filters also don't work

I think you are describing documented behavior (but this detail is not present in the manual page). Section 4.8.2 of the info page here says:

   * All paths are accepted _by default_, unless an exclusion filter
     says otherwise.

   * Inclusion filters may override any other exclusion filter.

   * The order in the definition of filters in the command line has no
     effect.

This explains why "exclude everything" (eg -e . or -e '.*') is a common pattern (see first two posts in this thread).

The change that I cited means that subsequent includes won't work unless the argument given on the command line (eg top-level directory) also matches an include flag. And often that's not the case, because we just want to look for events on *.ics files in some directory with an unrelated name like calendar_files/ or ./.

[xxxserxxx]

I missed the documentation.

Just to summarize what I understand: I see that if you do an exclude on a specific file name, but then an include on a variation of that file name, exclude+include works. Where it fails is when the exclude is only a wildcard, such as '.*' because -- as you say -- it's excluding the path itself, right?

The inclusion rule, even if documented, seems odd and particularly useless; it's a flag that is useless by itself, right? Or is there a case when -i by itself isn't essentially unused?

[mcarpenter]

The man page certainly needs some help ("the first matching expression wins" is misleading).

if you do an exclude on a specific file name, but then an include on a variation of that file name, exclude+include works.

I agree (inclusion wins over exclusion).

Where it fails is when the exclude is only a wildcard, such as '.*' because -- as you say -- it's excluding the path itself, right?

Right. This is the special case where the path to be watched is (now) ignored (before any watches get added even) because it matches the exclusion pattern, but not the inclusion pattern.

The inclusion rule, even if documented, seems odd and particularly useless; it's a flag that is useless by itself, right?

Given the default is "include", I agree: I don't see a use case for -i/-I without -e/-E.

[dinvlad]

Is there any update on this? Or any workaround to watch for files only for a particular extension on Linux?

(I guess one workaround could be to use fswatch **/*.ext, but that doesn't take into account newly created files)

[robfordww]

Hi, I was hit by the same bug as well on Ubuntu 20.04 LTS. This seems like a severe bug to me, that should be looked into.

[wcbdata]

Still seeing this bug at least in fswatch 1.14.0. If monitoring a whole directory, directly or recursively, using exclude and include .ext fails to detect anything because the search path (the directory) cannot match the include regex (*.ext).

I'm not sure whether this is fixed in a later version, but apt package managers seem to only have up through 1.14.0.

[moritztim]

I am using 1.17.1 and am still having this issue. This is a major problem! Please somebody fix this

Or any workaround to watch for files only for a particular extension on Linux?

You could use a regular expression to match everything but what you're looking for, and exclude that, I guess.

[moritztim]

I am using 1.17.1 and am still having this issue. This is a major problem! @dinvlad

Or any workaround to watch for files only for a particular extension

I guess you could write a regex for anything but what you're looking for, and exclude that

[aviloff]

Also here. I have a Makefile using fswatch. Last successfully used some time last year. I subsequently upgraded to ubuntu 20 and it no longer works, with symptoms as above.

Ubuntu 20 has fswatch 1.14.0. Launchpad indicates that ubuntu 18 had 1.11.2.

I rebuilt 1.11.3 (last release of minor version 1.11) and installed into /opt, and that works fine.

Conclusion: either a regression in fswatch >1.11.3 or a problem in the Ubuntu 20 packaging.

I had to build and install version 1.11.3 on a Raspberry Pi armv7 Raspbian to fix the same issue. Would be great if this thread was addressed in the next releases, and that those make it to apt-get!