Linux OS with .NET- The configured user limit (128) on the number of inotify instances has been reached

[travisgosselin]

This issue is directly related to and a followup of https://github.com/dotnet/aspnetcore/issues/7531 since that issue has been locked. The same issue occurs now in .NET Core 3.1. Following the previous thread, I was able to specify a different user in my Dockerfile to limit this problem which occurs when putting too many docker containers on the same host that are using the same user UID. Our teams now use a randomized UID to ensure each immutable build has its own ID:

RUN useradd --uid $(shuf -i 2000-65000 -n 1) app
USER app

This works to limit the issue, but the problem still continues if you intend to run the same instance of that immutable container with 9 or more instances on the same container orchestrator host. For example if I use the following image: "FROM mcr.microsoft.com/dotnet/core/aspnet:3." to run a base default / empty ASP.NET Core application with the following Program.cs (and usage of zero-configuration files):

Host.CreateDefaultBuilder(args)
                .ConfigureWebHost(builder =>
                {
                    builder
                        .UseKestrel()
                        .UseStartup<Startup>();
                });

Run a default app in the container above, and attach your shell and execute the following to count the number of watchers:

apt-get install lsof
lsof | grep inotify

.NET Core 3.1 shows me generally 15 or 16, while .NET Core 2.1 default app shows more like 40. If I am running 8 containers each taking at least 15 instances of the 128 limit - I run out pretty quickly.

General concerns:

1. Scaling is the first concern I have for obvious reasons (depending on my placement strategy). Is the expectation that I should be increasing this limit on my orchestrator hosts to allow for this?
2. With a default execution of ASP.NET Core 3.1 that uses no configuration files, why am I seeing 15 instances? Typically speaking in a container there is very little reason I even want to watch files (especially not configuration files) since its entirely immutable. Is there a way to entirely turn off the inotify watcher instances all together (which would solve this problem entirely)? Or is there an underlying bug in ASP.NET Core here?

EDIT by @Rick-Anderson #28387 mitigates the problem.

---

Associated WorkItem - 64021

[odin88]

+1 I face the same issue.

[johnnyreilly]

I'm also bumping hard on this issue; mostly in the context of being able to run integration tests inside a devcontainer. After a certain number tests in my project have been run I start to experience this problem. I'd love for this issue to be solved generally in dot net core.

I do wonder if it might be possible for Host.CreateDefaultBuilder to have an overload which did not use reloadOnChange: true and instead using reloadOnChange: false. If that was in place then people would be able to pivot easily to at least using less watchers.

The ideal solution is the underlying issue being fixed though :smile:

https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/generic-host?view=aspnetcore-3.1

[travisgosselin]

Thanks for jumping in @johnnyreilly - yes we are experiencing in our dev containers as well during builds. It can be recreated just based on the shared host. So it could be a build agent (single host) running many tests from different projects that hit this problem randomly, or it can be a single project WITH A LOT OF TESTS. frustrating to have to worry about this issue horizontally and vertically as well. Still trying to figure out what to do next... and additionally wondering why there are any default file watchers that I cannot disable, especially running in an immutable container in most instances....

Started to blog this journey a bit for our organization here: http://blog.travisgosselin.com/configured-user-limit-inotify-instances/

[BrennanConroy]

We should look into adding a doc about ulimit on mac and linux.

Also, mention using the polling file watcher to reduce inotify instances.

[Rick-Anderson]

@travisgosselin what doc should this go in?

[travisgosselin]

Good question @Rick-Anderson - Ideally it wouldn't be a concern at all and be fixed, but for the Docs specifically, it seems prudent to call out these best practices out in any location that discusses Dockerfile usage patterns for ASP.NET Core. Existing, "The Dockerfile" section here might be a starting place; it already indicates COPY of csproj and sln files as a good practice for docker layer caching (https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/docker/building-net-docker-images?view=aspnetcore-3.1). Calling out additionally some of the security practices described here for not running as root and the side effects from a port security perspective (i.e. has to be above port 1000) might make sense?

Perhaps here as well? https://docs.microsoft.com/en-us/dotnet/core/docker/build-container?tabs=windows Or perhaps this belongs closer to security? A new page as a sibling or close to something like this related to hosting: https://docs.microsoft.com/en-us/aspnet/core/security/docker-https?view=aspnetcore-3.1

[vasnab]

+1

[Rick-Anderson]

@BrennanConroy @travisgosselin several places report fixing this via reloadOnChange: false);

.AddJsonFile($"appsettings.json", optional: true, reloadOnChange: false);

See https://stackoverflow.com/a/56547116/502537

[BrennanConroy]

Sure, that could be a solution if you don't care about refreshing config on file change.

[Rick-Anderson]

@travisgosselin the following works in most cases:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureAppConfiguration((hostingContext, config) =>
        {
            config.AddJsonFile("appsettings.json",
                optional: true,
                reloadOnChange: false);
        })
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        });
}

See https://stackoverflow.com/a/56547116/502537 which is incorrect, CreateDefaultBuilder doesn't override reloadOnChange: false as last one set wins.

[Rick-Anderson]

Per @paillave here

echo fs.inotify.max_user_instances=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p this will increase the amount of authorized instances as too many may be taken by vs code addons

[travisgosselin]

Thanks for getting back to this @Rick-Anderson . All the suggestions you provide since my last post definitely help, and are actions I've taken across our services. That being said, even with my earlier investigation and initial notes, there seems to be no option to eliminate all watchers completely, even when setting reloadOnChange false. Generally speaking, I have that disabled anyhow, since this is inside a container, I'm not sure how useful it is. At the end of the day, all this information should be documented (if not fixed) indicating that you should NOT use the root user as a best practice, and the possible implications of scaling your service in PROD resulting in possible startup failure (if strategy places too many on a large host).

[johnnyreilly]

We're also bumping on this issue with .NET Core 3.1 and Function apps.

[Rick-Anderson]

See this comment by @Tratcher

[kanadaj]

@Tratcher would it be unreasonable to expect the Docker image to have DOTNET_HOSTBUILDER__RELOADCONFIGONCHANGE=false by default?

[Tratcher]

@kanadaj because you don't expect the config files to change without at least restarting? Seems reasonable.

@bradygaster @sebastienros do you know where to file that?

[sebastienros]

Should it be in the VS docker templates? Like we asked to change the App Service setting for forward proxy headers?

[johnnyreilly]

This sounds like a good idea. I've an issue related to this open on that repo here: https://github.com/microsoft/vscode-dev-containers/issues/418

Feels like this could be a PR!

[blackie1019]

@Tratcher would it be unreasonable to expect the Docker image to have DOTNET_HOSTBUILDER__RELOADCONFIGONCHANGE=false by default?

Totally agreed with this suggestion.

Because not all linux or emulators support this feature and this is not an essential function to make ASP.NET Core running in any version(for .NET Core 2 to .NET 6), it's absolutely turn to false by default to reduce redundant code put into the non sense block without the essential technical support.

Considering the QEMU not support this, and Apple M1 + .NET Core with Linux-x64 container running on this emulator I strongly recommend put this setting to False by default to help more rookie of ASP.NET Core without the pointless survey/research to play and learn ASP.NET Core on both Intel and ARM CPU on macOS system.

[Rombersoft]

I have the same problem on NET 6. I don't use docker and run the application on a Linux host. But my error happens in the code:

WebApplicationBuilder builder = WebApplication.CreateBuilder();

Exception is:

System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.ConfigurationExtensions.Add[TSource](IConfigurationBuilder builder, Action`1 configureSource)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(IConfigurationBuilder builder, IFileProvider provider, String path, Boolean optional, Boolean reloadOnChange)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(IConfigurationBuilder builder, String path, Boolean optional, Boolean reloadOnChange)
   at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.<>c__DisplayClass11_0.<ConfigureDefaults>b__1(HostBuilderContext hostingContext, IConfigurationBuilder config)
   at Microsoft.AspNetCore.Hosting.BootstrapHostBuilder.RunDefaultCallbacks(ConfigurationManager configuration, HostBuilder innerBuilder)
   at Microsoft.AspNetCore.Builder.WebApplicationBuilder..ctor(WebApplicationOptions options, Action`1 configureDefaults)
   at Microsoft.AspNetCore.Builder.WebApplication.CreateBuilder()

Is any suggestion?

[Rombersoft]

I understood that this problem is in VS Code. So I have 3 cases:

1. VS Code is loaded and Rider IDE is unloaded: my application has this exception when I run it from VS Code.
2. VS Code & Rider are loaded and have opened the same solution (.sln): my application has this exception when I run it from any IDE, it does not mean VS Code or Rider.
3. VS Code is unloaded and Rider IDE is loaded; my application does not have this exception!!! WTF???

I see that VS Code offer me to create launch.json file:

{
    // Используйте IntelliSense, чтобы узнать о возможных атрибутах.
    // Наведите указатель мыши, чтобы просмотреть описания существующих атрибутов.
    // Для получения дополнительной информации посетите: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": ".NET Core Launch (web)",
            "type": "coreclr",
            "request": "launch",
            "preLaunchTask": "build",
            "program": "${workspaceFolder}/GUI/TerminalUI/bin/Debug/net6.0/TerminalUI.dll",
            "args": [],
            "cwd": "${workspaceFolder}/GUI/TerminalUI",
            "stopAtEntry": false,
            "serverReadyAction": {
                "action": "openExternally",
                "pattern": "\\bNow listening on:\\s+(https?://\\S+)"
            },
            "env": {
                "ASPNETCORE_ENVIRONMENT": "Development"
            },
            "sourceFileMap": {
                "/Views": "${workspaceFolder}/Views"
            }
        },
        {
            "name": ".NET Core Attach",
            "type": "coreclr",
            "request": "attach"
        }
    ]
}

On another website I read that VS Code runs the application on docker but I don't see where in launch.json it is directed that application must run in docker

[smorokin]

On another website I read that VS Code runs the application on docker but I don't see where in launch.json it is directed that application must run in docker

@Rombersoft: This is only true if you use the dev container feature of it. You would have a .devcontainer folder in your project if it is the case

[alexkreskiyan]

In my case of simple linux host (no problem with containers yet), handled the problem with: https://github.com/mikesart/inotify-info - to identify inotify consumers Updated /etc/sysctl.conf with:

fs.inotify.max_user_instances=1024
fs.inotify.max_user_watches=1048576

Important to note, initially problem happened due to limit of user instances, set to 128, not limit of user watches

[Rick-Anderson]

@travisgosselin see what @alexkreskiyan has above this. Looks useful.

[travisgosselin]

@travisgosselin see what @alexkreskiyan has above this. Looks useful.

Sure, I would imagine that additional limit would be important as well, though I haven't run into that myself specifically. I have increased the user_watches on our systems as a workaround in my case.

[andrewhornsby-bjss]

We had this problem running tests on our devops linux self-hosted build agent. Interestingly we didn't get the issue on the the cloud agent, so maybe it is configured differently.

The above fix has solved the problem for us though. Thanks @alexkreskiyan!

[LanAnh002]

I have same problem but seem can't not resolve: blazor.webassembly.js?v=1:1 crit: Microsoft.AspNetCore.Components.WebAssembly.Rendering.WebAssemblyRenderer[100] Unhandled exception rendering component: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached. I try to fix this issue by four way:

1. change config: reloadOnChange => false
2. Change variable environment DOTNET_USE_POLLING_FILE_WATCHER => true
3. Change variable environment DOTNET_HOSTBUILDER__RELOADCONFIGONCHANGE=> false
4. Change max user instance: ・echo fs.inotify.max_user_instances=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p ・echo 512 > /proc/sys/fs/inotify/max_user_instances => It's not work

[lonix1]

Just a note that the workaround by alexkreskiyan is a temporary fix. What you'll quickly find is that every few months you're going to bump those values higher and higher. At some point that will impact your machine's performance.

If this is in development it's fine (I have to set those values sky high to be able to use vscode + c# effectively). But be wary when using that workaround in production.

There's an underlying .NET problem that creates this problem in the first place. That closed issue mentioned in the OP should not have been closed/locked.

[macias]

@LanAnh002 do you use step (2) this way? I put this right at the start of my program and I didn't hit this bug in a long time.

        Environment.SetEnvironmentVariable("DOTNET_hostBuilder:reloadConfigOnChange", "false");
        var builder = WebApplication.CreateBuilder(args);

[LanAnh002]

@macias Your way seem like my way 1:

Host.CreateDefaultBuilder(args) .ConfigureWebHostDefaults(webBuilder => { var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT"); var isDevelopment = environment == Environments.Development; string settingFile = "appsettings.json"; var builder = new ConfigurationBuilder() .SetBasePath(Directory.GetCurrentDirectory()) .AddJsonFile(settingFile, optional: true, reloadOnChange: false); IConfigurationRoot configuration = builder.Build(); var runUrl = configuration.GetValue("MySettings:RunUrl"); webBuilder.UseStartup(); webBuilder.UseUrls(runUrl); })

[macias]

@LanAnh002 it will cost you nothing, could you please put "my" first line as your first one? Explanation -- it is better to keep it global, because deep down in .Net code the default mode is to reload, and it is a bit pointless relying on .Net it won't open this or that file in this mode.

[LanAnh002]

@macias Thank for your suggestion. I try add your first line as my first one but it didn't work. I check that my project use this just not in Program.cs, So when i chang all this config: reloadOnChange => false, it work well.