Linux: implement a different watcher that does not require file handles per file in the workspace folder

[GregoryLundberg]

• VSCode Version: 1.21.0
• OS Version: Arch Linux

Steps to Reproduce:

1. Run 'code' from command line or click menu item to run it.

Does this issue occur when all extensions are disabled?: Yes

Dialog message indicates the web page has help. It does not.

Bumped (soft) open file limit from 1024 to 4096: no help.

[Stigjb]

I just got this myself when opening a new folder in a window I had open. I clicked the button to see instructions, but nothing happened.

VS Code version: 1.21.0 OS version: Linux Mint 18.2 Sonya 64-bit.

[bpasero]

@GregoryLundberg @Stigjb the button should open a browser pointing to these instructions: https://code.visualstudio.com/docs/setup/linux#_error-enospc

Did no browser open?

[GregoryLundberg]

It opened for me, but there is nothing there about running out of file handles, only about running out of disk space. Neither are true and I clicked to never see the notification again because it's a obviously a false-positive.

[bpasero]

@GregoryLundberg so you did not see https://code.visualstudio.com/docs/setup/linux#_error-enospc open in your browser? What page did open for you then?

[GregoryLundberg]

YES the browser opened. YES it showed that page. NO it does not speak about how to fix your program bug claiming it ran out of FILE handles.

ENOSPC means out of space on the drive. I am not.

The notification says it's out of FILE handles. It is not. I increased the limit from 1024 to 4096 to no effect.

As I read the web page, it's about running out of some other sort of handles so it's not about FILE handles and does not apply to the notification.

I clicked to ignore the error because it's obviously a false positive since there are plenty of file handles.

I thought perhaps I could re-enable it to post a screen shot but I don't see a settings option for it. Where did you hide the options? There doesn't look to be anything under File|Preferences|Settings or <gear>|Settings.

[GregoryLundberg]

OK I read what that page says and half a million file "watches" won't do. If it's really a problem you'll either need to clean up your code or document that VS Code is only for small to moderate sized projects.

My project tree has far more than half-a-million files, and there is no reason for you to "watch" every single one of them, which is, I guess, what that error is about.

I guess your notification meant "out of file-watch handles" and someone omitted the qualifier.

Perhaps the breakage is you're trying to watch everything when you really only need to watch files which are actually open in the editor (which, at the time the notification appeared was one: your release notes, which automatically appears).

[Stigjb]

The browser did not open for me.

[bpasero]

@GregoryLundberg we should probably update the documentation so that its clear that the ENOSPC error is in fact the same error for running out of file handles.

Currently we have no other way than requiring you to change the limit in your config file to make it work.

@Stigjb that is weird. if you have a http link in the editor and you ctrl+click it, does it also not open any browser?

[GregoryLundberg]

You're serious? What in the world do you need more than 4096 open files for? If you're trying to open every file in my project, I doubt I can set the limit that high.

I suppose I could try bumping to 32K. It won't come close to opening every file, but it sounds sufficiently insane. Not that I can tell if it has an effect because I don't know how to re-enable the notification, so it seems sort of pointless.

[bpasero]

@GregoryLundberg two things we can try. First, we have a setting "files.useExperimentalFileWatcher", can you try setting that to true, restart Code and then see if it happens still?

[GregoryLundberg]

How do I re-enable the notification? Not much point in testing if I can't see the results.

[bpasero]

@GregoryLundberg that notification shows up per workspace, so if you just rename the folder you open it should show again.

[GregoryLundberg]

Sorry. No can do.

[bpasero]

@GregoryLundberg no problemo, an easier way of doing it: from the command line run code --user-data-dir <some folder>. This will open a fresh version of Code with no prior state. That also means that you will have to change the setting I mentioned in this case again and restart once.

[GregoryLundberg]

Too bad there's no way to re-enable the notifications properly. Probably should open a new issue for that. Seems like a pretty serious design flaw.

Using a forced directory, I see the notification, again. I switched to your experimental watcher and still see the notification. I set the max open file handles to the limit of 4096 (can't set it higher) and still see the notification.

Are you sure the message is not incorrect and it's not about file handles at all? The documentation says file WATCH handles. Maybe the documentation is correct and the notification message is wrong?

Well, whatever is wrong, it seems the correct option is to ignore it. You don't seem to need whatever you're complaining about.

[bpasero]

For resetting the notification preference, we have https://github.com/Microsoft/vscode/issues/24815

The message is correct and it is about file handles. The file watcher we have needs 1 file handle per file on Linux to support the watching (unfortunately). When the list of file handles exceeds the OS limit, VS Code will still function but some file events will not be recorded and be missing. E.g. if you delete a file in the Linux explorer, you will not see that file go away from the VS Code explorer.

Another thing to try is to configure files.watcherExclude setting. The idea is that you can exclude parts of the workspace from watching that contain many files. You can do that back when you run normally and when you set back to "files.useExperimentalFileWatcher: false".

Is there a particular large folder that you hide, e.g. some out folder with compiled contents?

[GregoryLundberg]

So problem is simply bad design on your part.

Since you use one file handle per file, you have a limit on project size of 4096 files.

You should clearly document, therefore, that VS Code is only usable for small projects. Since my project has over 8000 files just for the primary C++ headers and footers, there is no way for you to work without error.

I've not used inotify(7) but would have assumed it was more efficient than you're implying.

[bpasero]

@GregoryLundberg yeah it is bad design on our part, we should only watch what is actually needed by having an API where you opt-in to specific paths that should be watched as opposed to watching everything.

I do not think there is a way to watch files on Linux recursively without requiring a file-handle per file in the workspace. On Windows and macOS however, file watchers do NOT require any file handles per file. So this issue is actually only showing up on Linux unfortunately.

[GregoryLundberg]

According to the man page, one file-watcher handle per directory should do for all files and directories within that directory. It does not say recursive, so, I'd assume (not tested) that creating, renaming or deleting a directory would get a notification event on the directory where it appears.

As I said, though, I've never actually used inotify and am not sure I could throw together a test case.

[sagebind]

I'm also noticing this. I'm much more curious to hear why this is suddenly a new problem. Why did 1.20 not complain about file handles? Was it just silently ignored? I've used Code on workspaces with 10,000+ files before with no obvious issues or loss in functionality.

[bpasero]

@sagebind yes it was silently ignored and you would see no file watching support without actually understanding what is going on.

[HankB]

I think I'm running into the same issue. On startup I get a warning "Visual Studio Code is running out of file handles. Please follow the instructions link to resolve this issue" and the "instructions" link goes to https://code.visualstudio.com/docs/setup/linux#_visual-studio-code-is-unable-to-watch-for-file-changes-in-this-large-workspace-error-enospc I see the following:

hbarta@yggdrasil:~/Documents/shell/shell_scripts$ cat /proc/sys/fs/inotify/max_user_watches
65536
hbarta@yggdrasil:~/Documents/shell/shell_scripts$ find .|wc -l
256

I've opened code with a single file which is a shell script. I don't have any extensions for this file. I've disable all installed extensions and restarted code and get the same warning.

The file browser window starts with my home directory. I hope it is not trying to watch all personal files on the PC. I thought the last time I looked, the file directory tree started with the current directory when opened.

Applying the fix from the web page eliminates the warning but it seems odd that code requires that many handles to monitor files in a working directory that has 256 files (mostly from git.) There are actually 13 project files and no subdirectories (aside from .git.)

Correction, the fix does not work. It just takes longer for the warning to show up. This is on a fairly powerful laptop:

Lenovo Y-50
I7-4710HQ
16GB RAM
SATA SSD (Samsung 850 EVO 1TB)
Debian 9 (Stretch)

During this time 'top' reports over 100% CPU usage and about 12% memory usage. After a some time (5-10 minutes I think) the CPU usage and RAM usage both drop to a couple percent.

[bpasero]

@HankB we meanwhile fixed this issue that we would open your home directory. If you do not want to wait for the fix to reach stable, you can use our insiders release: You can give our preview releases a try from: https://code.visualstudio.com/insiders/

[Stigjb]

@bpasero A bit late to answer ­­­­– no, HTTP links in files do not open for me when I Ctrl+click. So it must probably be unrelated to this issue.

[thierer]

@bpasero thanks for clarifying that it's a new notification. I also wondered why I never got that message with prior versions of vscode. That said, I now also regret disabling the warning. Is there a practical way to manually reset it before #24815 gets resolved? Permanently renaming the directory isn't really a viable solution for me neither.

Edit:

Never mind, just found the setting with a little help from the instructions in #23314's discussion...

As a reference for others, the setting is storage://workspace/root:<some uid>/ignoreenospcerror

[otono]

For me works solution from this page https://code.visualstudio.com/docs/setup/linux#_error-enospc

When you see this notification, it indicates that the VS Code file watcher is running out of handles because the workspace is large and contains many files.

The limit can be increased to its maximum by editing /etc/sysctl.conf and adding this line to the end of the file:

fs.inotify.max_user_watches=524288

The new value can then be loaded in by running sudo sysctl -p.

[GregoryLundberg]

The 'solution' on the web page is just a hack to hide the problem.

Any solution requiring sudo is sub-optimal. It presumes that the reader actually has the required privileges, which may not be the case.

A true solution would be to improve efficiency by, for example, watching entire directories, instead of individual files. (OK, this won't completely solve the issue but it moves it far, far away before it re-appears.)

An alternative would be to fall back to the old tried-and-true method of re-scanning the entire project every now and then.

And, for those of us who know our projects don't add/remove/rename while we're using Code (or know precisely when they do), falling back to disabling automatic scans and only using manual scans, is marginally acceptable.

This is the point we're sort-of at now: if your project is large enough to tickle this bug, and you don't or won't hack your system (or the hack won't help), disabling the message so we don't have to hear the feature has been disabled, and manually re-starting Code when we need a re-scan is only slightly more hassle than having a UI feature for the operation.

So, I see three issues:

• the error message is wrong. It is not file handles which have exhausted, it is file-watcher handles: a different thing entirely.

• once suppressed, there is no way (other than mucking around in Code internals or renaming your project's directory) to re-enable the notification.

• the file-watching implementation is deficient.

I believe I've seen action on some of these.

[4creators]

The 'solution' on the web page is just a hack to hide the problem.

Any solution requiring sudo is sub-optimal. It presumes that the reader actually has the required privileges, which may not be the case.

This is not the question about sub-optimal solution or sudo usage it is installer bug. Since quite a bit of ppl are using VS Code to work on MSFT repos like coreclr and corefx they hit this problem first time thay update to newest version. It should be labeled as bug or installer bug and fixed asap.

[d1sco]

just got this error but ignored it, best solution for seems to hide it like GL suggests

[MaelAbgrall]

hello, so I have the same issue but i'm not using as much files as you: I've 600 images in a subfolder, a virtualenv & my scripts

this issue however does not happen when i'm opening folders with only a few files (<10)

I'm on Fedora 26 On windows 10, I don't have this issue

[caub]

I've the same running out of file handles error, in a directory with 25510 files in total (most are in node_modules/**, even if files.watcherExclude contains node_modules/**' by default, it's not helping). The big issue is that webpack hot-reload doesn't work anymore when vscode is running, because vscode consumes all the file watchers.

Increasing watches with fs.inotify.max_user_watches = 131072 didn't fix it. Anyway I'd prefer a more solid way like mentioned in this thread

[markonyango]

I have run into this problem as well. I use nodemon in my environment to restart a nodeserver. Since VS Code hogs all the filewatch handles all that's left to me is to manually start and execute my gulp and nodemon tasks. Are you sure that the re to ignore node_modules works as it is supposed to? Once I added node_modules to files.exclude and restarted, I was able to start nodemon again from VS Codes internal terminal.

EDIT: Scratch that solution. If you let VS Code run for some time, nodemon will start complaining again, leading me to believe that this is not an issue of watching node_modules but something else.

EDIT 2: HOWEVER it seems to work again if you add

"**/node_modules": true

to files.watcherExclude. The preset option

"**/node_modules/**": true

seems to do nothing. This is obviously not a fix for anybody that has many files in their project that do not come from a node_modules folder. It's interesting though that the preset does not what it is supposed to do.

[Teebo]

On this link search for 'ENOSPC', you will find the directions to resolve this issue or have a quick read at Increasing the amount of inotify watchers, hope this helps

[jasonbeach]

Just wanted to chime in as well. It would be nice if you could tell vscode to ignore certain directories in your workspace such as a build directory. In one python project I have, one directory has several thousand images (image tiles for mapping) which vscode doesn't need to know anything about. Moving that folder outside of the workspace would be clumsy.

[caub]

@jasonbeach there's files.watcherExclude, search.exclude, files.exclude that probably fit your need

@Teebo we know that, thanks but it's not the best way to handle it, and doesn't work well

@markonyango's thanks, what you said seems to work so far, since a week or so I tried

[skylize]

@caub

even if files.watcherExclude contains node_modules/**' by default, it's not helping

The glob pattern for watcherExclude needs to be an absolute path. Add some stars to the front of your glob to get it to ignore all node modules.

"files.watchExclude":  {
   "**/node_modules/**": true
}
`

[dotnetCarpenter]

@markonyango nailed it! This appears to be a bug in the handling code of files.watcherExclude pref.

"**/node_modules/**": true, is a correct globing pattern but it does not work.

My file structure is:

project root
|── frontend/
|   └── node_modules/
|── server/
|   └── node_modules/

Running:

echo "MAX file watches allowed on system"
cat /proc/sys/fs/inotify/max_user_watches

echo "Current files in project - excluding node_modules"
find . -type f -not -path '**/node_modules/**' | wc -l

echo "Current files in project - including node_modules"
find . -type f | wc -l

outputs:

MAX file watches allowed on system
8192
Current files in project - excluding node_modules
149
Current files in project - including node_modules
31705

Current workaround is to add the following to your user settings:

{
    "files.watcherExclude": {
        "**/.git/objects/**": true,
        "**/.git/subtree-cache/**": true,
        "**/node_modules/**": true,
        "**/node_modules": true
    }
}

Thanks to @markonyango

vscode: 1.25.1 1dfc5e557209371715f655691b1235b6b26a06be x64 os: Linux dotnet-Aspire-ES1-311 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[acrixl]

Current workaround is to add the following to your user settings:

this solution not work for me

vscode: Code - OSS Version: 1.25.1 Commit: 1dfc5e557209371715f655691b1235b6b26a06be Date: 2018-07-26T23:50:00.463Z Electron: 1.7.12 Chrome: 58.0.3029.110 Node.js: 7.9.0 V8: 5.8.283.38 Architecture: x64

debian buster/sid

[AdrienHorgnies]

I encountered the same bug. It doesn't exclude the node_modules folder.

There should be only 55 files to watch, well under the limit :

$ find . -not -path "**/node_modules/**" -not -path "**/.git/objects/**" -not -path "**/.git/subtree-cache/**" -type f | wc -l
55

But well over the limit if you don't exclude files :

$ find . -type f | wc -l
31596

I'm sharing my project if you want to reproduce the bug. It is an Angular project from a tutorial. The resources are available for free on the official site but you are supposed to create an account... So for your convenience, I uploaded a 90kB archive.

The easiest way to trigger the bug is :

rm -r node_modules/*
npm install

Version: 1.25.1 Commit: 1dfc5e557209371715f655691b1235b6b26a06be Date: 2018-07-11T15:40:20.190Z Electron: 1.7.12 Chrome: 58.0.3029.110 Node.js: 7.9.0 V8: 5.8.283.38 Architecture: x64 O.S.: Ubuntu 18.04.1 LTS

I've added the following to my user settings to work around :

{
    "files.watcherExclude": {
        "./.git/objects/**": true,
        "./.git/subtree-cache/**": true,
        "./node_modules/**": true
    }
}

[GregoryLundberg]

Just a reminder: the problem has nothing to do with Node or the NPM. The recent discussions involving Node are just easy examples of a more general issue. Some of us cannot use the work-around of excluding Node modules because we don't use Node in our projects. Some of us have projects which exceed the limit due to the sheer number of first-class assets. For us, as nice as this feature might be, the only viable option is to completely disable it.

[acrixl]

also good idea to stop use node and npm this solve problem with node_modules

[AdrienHorgnies]

I've come from an issue which was marked as duplicated of this issue. So I thought it was pertinent to post here. I agree that it doesn't concern nodejs or npm but as you said, it is an easy example and it happens to be the problem I have. I'd like to stop using node and npm but I've come to the conclusion that all package/dependency managers suffer similar problems.

[acrixl]

i like to start package/dependency managers holy-war, but here more interesting stuffs is someone can check windows platform and insiders build have same problems?

[ghost]

@bpasero, re: https://github.com/Microsoft/vscode/issues/45295#issuecomment-372676016

Hi, using kubuntu 18.0.4 with latest kernel (4.18 rc) I am having this issue still, with insiders release 1.26-insider

code -v

1.25.1
1dfc5e557209371715f655691b1235b6b26a06be
x64

On multiple diff workspaces, confirmed via this command that there aren't too many in the workspace:

$ find . -not -path "**/node_modules/**" -not -path "**/.git/objects/**" -not -path "**/.git/subtree-cache/**" -type f | wc -l
3625

[dotnetCarpenter]

This doesn't appear to be a linux only issue.

40898 is the exact same issue but on Windows 10.

"files.watcherExclude" just does not work...

[dotnetCarpenter]

If anyone is up for fixing this, here is a good overview of which files that needs to be debugged: https://github.com/Microsoft/vscode/search?q=%22files.watcherExclude%22&unscoped_q=%22files.watcherExclude%22

The glob patterns seems to be the culprit - e.i. they do not match the correct paths.

For linux/unix, the ignored variable should be examined: https://github.com/Microsoft/vscode/blob/master/src/vs/workbench/services/files/node/watcher/unix/watcherService.ts#L111

[GregoryLundberg]

I fail to see how fixing a bug in the glob handling fixes the issue, All it seems to do is fix the workaround for those for whom the workaround might help. Am I missing something?

[dotnetCarpenter]

@GregoryLundberg most people who have posted here, does not have too many files in their project folder(s) but the glob patterns doesn't work, so they get the issue with no possible work-aroud.

Say you wanted a workaround so vscode at least was useable to you and you could run other programs that watch files at the same time, you would need to set "files.watcherExclude" for your project.

Currently you can't. Fixing the glob patterns is a first step to fix your originale, 500000 files in a project, problem.

[lucafaggianelli]

Any update on this bug?

[Belar]

I run into watcher issues a few weeks ago, seemed exactly like the problem described in https://github.com/Microsoft/vscode/issues/50417 which led me to this issue and its references; all of which fit the pattern with node_module, watcherExclude and so on.

Here are 2 observations which I didn't see mention anywhere, although behind the scenes (just a guess, I'm not familiar with VS Code code base) they "feel" like related to multi-root workspace (https://github.com/Microsoft/vscode/issues/40898)

1. Noticed with VS Code's settings.json (when opening using "Open Settings (JSON)"); Opening file from outside the root of workspace breaks watcher
2. Opening different directory in a new window breaks watcher

If those are expected behaviors, then you can skip the rest of the post, which is reproduction steps.

Test objects are 2 clean projects (App1, App2) generated with create-react-app (npx create-react-app <project_name> is all that's needed). After running @dotnetCarpenter script from https://github.com/Microsoft/vscode/issues/45295#issuecomment-406810939 we get:

MAX file watches allowed
8192
Current files in project - excluding node_modules
52
Current files in project - including node_modules
27741

Max file watchers count is the default one, and each project, without node_modules, has 52 files.

First case

App1 can be opened in VSC without problems, npm start will run just fine (file changes will be recorded by VSC and the dev mode watcher). However, after opening e.g. settings.json with > Preferences: Open Settings (JSON) familiar pop-up shows up and npm start on App2 will render watcher error (that's theoretically on 104 files - non node_modules elements from App1 and App2). Running apps isn't necessary, a "reliable", minimal reproduction is to open App1, open src/App.js, open settings.json, do changes with saving to the src/App.js and switch tabs to settings.json - should suffice for pop-up to show up (although it's not 100%, more like 85), npm start will render watcher errors.

Second case

Open App1 in VSC, run it (order doesn't matter), changes will be recorded correctly e.g. correctly reflected in VSC's git tool. Open new VSC window (Ctrl + Shift + N), open App2. Familiar pop-up may (should, but not always) show up now, and if you try to run App2 (npm run start) there will be no watchers available.

When working in single directory and files within, the exclusion seems to be working as expected (at least I couldn't find a minimal reproduction; same when paying attention in real life projects). Also, didn't notice above issues (out of watchers when trying to run app in dev mode) with other editors.

I hope above helps at least a bit.

VS Code info at the time of writing:

• v1.29.1
• clean settings or "files.useExperimentalFileWatcher": false in global settings (which is default Linux, but in other reports it was suggested to set it explicitly)
• all extensions disabled