ignore/exclude directory in watchlist not work

[nirajvara]

I have configured the watchman 4.9.4 on Ubuntu and its working fine.

For Directory exclude/Ignore I have done the setting like below but it didn't exclude the directory

cat state { "watched": [ { "triggers": [ { "command": [ "/home/niraj/watchjson.sh" ], "append_files": true, "name": "jsontest", "ignore_dirs": [ "/home/niraj/jsontest/ignore" ], "stdin": [ "name", "exists", "new", "size", "mode" ] } ], "path": "/home/niraj/jsontest" }, ], "version": "4.9.4" }

As its not work then I added the .watchmanconfig

{ "ignore_dirs": ["/home/niraj/jsontest/ignore"] }

but its not work. Please guide how can I exclude directory/files.

[wez]

The paths must be relative to the watched root:

{
   "ignore_dirs": ["ignore"]
}

[nirajvara]

Dear Wez, Thank you. Its seems to work when configure in .watchmanconfig.

I want to set this in Json file and its not working. Please guide how to define this on my json file.

I have bold the Entry above that I have added in my json file.

[wez]

The state file is not meant to be something that you edit by hand. If you want watchman to ignore a directory then it must be specified in the .watchmanconfig file as I indicated above.

What is your higher level goal? What is it that you want to use watchman for?

[nirajvara]

Dear Wez,

My goal is I have nextcloud snap it has multiple users when any users upload files or modify files it should be backup to other location. Everything seems to work. I am not using the STATE file watching directory instead I have made my own json file and run it by this command
watchman -j < jsonfile Problem is here I have to add the .watchmanconfig file on every users home. if any method to add this ignore_dirs in that json file so I need to maintain only that json file. My Json file is here so I can add here the multiples user's directory for watch. This file is only sample file. ["trigger", "/home/niraj/jsontest", { "name": "jsontest", "stdin": [ "name", "exists", "new", "size", "mode" ], "command": ["/home/niraj/watchjson.sh"], "append_files": true }]

So Please guide If you have any idea to achieve the same.

[wez]

We don't have that sort of functionality today. https://github.com/facebook/watchman/issues/202 is an issue that raises the idea of making this sort of configuration more flexible, but we don't have a burning need for this and thus haven't implemented it.

Do you really need to use ignore_dirs? Can you tell me more about what you want to ignore? That might help me suggest alternative options.

[nirajvara]

In Nextcloud when you upload, the upload size is supposed 100 MB. then it creates a chunks of 10 MB in upload folder. 100 MB size so it will create 10 MB 10 chunks that is not required to be sync. so this directory upload need to be ignore as its used for chunks data only.

[wez]

How many files are likely to be in the upload folder? How frequently are they created and deleted? If the volume and rate of change is relatively low, then you don't need to use ignore_dirs and can instead exclude matching paths in your trigger query expression:

$ watchman -j <<-EOT
["trigger", "/home/niraj/jsontest", {
  "name": "jsontest",
  "expression": ["not", ["match", "upload/**", "wholename"]],
  "command": ["/home/niraj/watchjson.sh"],
  "stdin": ["name"],
}]
EOT

[FuriouslyCurious]

Hi @wez and @nirajvara I am experiencing a similar issue. Our use case is a medical DICOM server where we need to watch folders and trigger certain scripts for each patient image created / edited in watched folder.

Adding "ignore_dirs" directive in local .watchmanconfig file works as intended, but copy-pasting the same directive in global /etc/watchman.json file does not work.

[nirajvara]

How many files are likely to be in the upload folder? How frequently are they created and deleted? If the volume and rate of change is relatively low, then you don't need to use ignore_dirs and can instead exclude matching paths in your trigger query expression:

$ watchman -j <<-EOT
["trigger", "/home/niraj/jsontest", {
  "name": "jsontest",
  "expression": ["not", ["match", "upload/**", "wholename"]],
  "command": ["/home/niraj/watchjson.sh"],
  "stdin": ["name"],
}]
EOT

Hi

I tried the above expression but its works only when file modification is there. means only ignore the file modification but when created a new file in /upload, expression is not working.

[FuriouslyCurious]

@wez Figured it out based on your example above, thanks!

PS: (Suggestion in issue #463 might be a good idea as Watchman usage grows.)

[guilhermelepore]

I have the same problem. I need to exclude/ignore a .ssh directory in my watched root, which is /home/myuser/, so I added to the /home/myuser/.watchmanconfig file, the following lines:

{ "ignore_dirs": [".ssh"] }

But Watchman keeps executing the trigger for this directory.

[nirajvara]

Please try this

"expression": ["allof", ["match", "*"], ["not", ["match", ".ssh/**", "wholename"]]

[ArtemBernatskyy]

For those landing with similar issue try this

Check if watchman reads you config correctly watchman get-config .

If it didn't read it restart the daemon via watchman shutdown-server

[oblitum]

If I have my ~ under git, how do I prevent it from being watched by watch-project on some random subfolder whose only parent folder under VCS is ~?

I wish to have my ~ under git, but I don't want watchman to consider it a project, I wished it excluded, effectively skipping it. I was unable to find a way.

[wez]

@oblitum take a look at https://facebook.github.io/watchman/docs/config.html#root_files

I'd suggest putting this in /etc/watchman.json:

{
  "root_files": [".watchmanconfig"]
}

it will cause watchman to only allow watches for directory hierarchies that have a .watchmanconfig file at their root. You can then check a .watchmanconfig file containing {} (an empty config) into projects that you want to use with watchman.

This is how watchman is deployed within FB/Meta.

[oblitum]

Thanks for the info @wez. That seems reasonable, although it's just not as convenient. I think I'll follow the same, but at the same time, in the context of coc.nvim, I wished it could be automatically triggered in all my projects under git, except for dotfiles/homedir, for obvious reasons.

coc.nvim does try to prevent watchman from launching for this but it fails miserably, only avoiding it for file access from home folder directly, but not from subfolders (this is explained in the linked back issue).

[oblitum]

.watchmanconfig file containing {} (an empty config)

Would be nice if empty file also worked as empty config (it's easier to recall just a touch .watchmanconfig is needed to enable it, than an specifics like echo '{}' > .watchmanconfig).

[oblitum]

I'd suggest putting this in /etc/watchman.json:

{
  "root_files": [".watchmanconfig"]
}

I did that and weirdly enough it changed the behavior of cd ~/foo/bar && watchman watch-project $PWD from picking ~ as root due to " ~/.git" to "watching parent directory" (~/foo/bar) by default it seems. It seems convenient to have, but I wasn't expecting that, I was expecting it to not watch anything. It seems that's how watch-project is designed (indeed, of course, given "root_files" originally just had a longer list, but still operated the same when not finding any of such "root_files").

[wez]

Ah, you also want: https://facebook.github.io/watchman/docs/config.html#enforce_root_files

[oblitum]

Okay, thx. Well, to simplify my life at not hitting watchman's default watching excess (in my experience), I've put export WATCHMAN_CONFIG_FILE="$HOME/.config/watchman/watchman.json" in my dotfiles, with bellow contents and pushed it into my dotfiles repo.

{
    "enforce_root_files": true,
    "root_files": [".watchmanconfig"]
}

Thanks for support.

[oblitum]

Sigh... it seems this made me go back to square one, it seems a bug with watchman?

Projects/samples/c++ on @ archlinux [!]
❯ watchman watch-project /home/francisco/Projects/samples/c++
{
    "watch": "/home/francisco",
    "watcher": "inotify",
    "version": "20211017.225819.0",
    "warning": "opendir(/home/francisco/.local/share/containers/storage/overlay/9f49eb96f6023f08eb2c28582195a5137fad015205e4742605d98d649959ec43/diff/var/lib/postgres/data) -> Permission denied. Marking this portion of the tree deleted\nTo clear this warning, run:\n`watchman watch-del '/home/francisco' ; watchman watch-project '/home/francisco'`\n",
    "relative_path": "Projects/samples/c++"
}

Projects/samples/c++ on @ archlinux [!] took 17s
❯ watchman watch-list
{
    "version": "20211017.225819.0",
    "roots": [
        "/home/francisco"
    ]
}

Projects/samples/c++ on @ archlinux [!]
❯ echo $WATCHMAN_CONFIG_FILE
/home/francisco/.config/watchman/watchman.json

Projects/samples/c++ on @ archlinux [!]
❯ bat $WATCHMAN_CONFIG_FILE
───────┬─────────────────────────────────────────────────────
       │ File: /home/francisco/.config/watchman/watchman.json
───────┼─────────────────────────────────────────────────────
   1   │ {
   2   │     "enforce_root_files": true,
   3   │     "root_files": [".watchmanconfig"]
   4   │ }
───────┴─────────────────────────────────────────────────────

The thing works when put on /etc/watchman.json, but not from $WATCHMAN_CONFIG_FILE. It seems it's being simply ignored.

[wez]

You need to make sure that $WATCHMAN_CONFIG_FILE is set when the server is first spawned.

Additionally, if a watch is already persisted in the state file, then the restriction will be ignored when re-loading that watch. I'd suggest running watchman watch-del-all to clear all of those out.

[oblitum]

I'd suggest running watchman watch-del-all to clear all of those out.

I did!

Consider I always do watchman watch-del-all && watchman shutdown-server before trying stuff out.

[oblitum]

Maybe my build has not been compiled with this flag?

But alas, using this undocumented ~/.watchman.json solved my problems! It's what I'm committing to my repo now, instead of something based on the environment variable.

[oblitum]

Source is truth :)

[oblitum]

Hey. Just like to report back that I think the real fix for my problem is that watchman didn't had watchman watch deprecated, because its own (watch-)project heuristic might be irrelevant when watchman is used as a dependency of some tool that already have its own heuristic for project roots. Both watchman watch-project and watchman watch have their own use-cases.

https://github.com/neoclide/coc.nvim/issues/3375#issuecomment-971846143