System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.

[Gprsk]

I've been getting this issue regularly when using Docker on a Kubernetes deployment. Code did not change, but after a while we are unable to start new containers because they keep crashing.

My setup code is as simple as it can be:

        public static void Main(string[] args)
        {
            Console.WriteLine("API Initializing...");

            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args)
        {
            IHostBuilder host = 
                Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                });
            return host;
        }

Startup class is quite simple:

    public class Startup
    {
        public static Container __injectorContainer__ { get; } = new Container();
        public static bool __simpleInjectorVerificationInsideScope__ = true;

        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddControllers();

            services.AddSimpleInjector(__injectorContainer__, options =>
            {
                options
                    .AddAspNetCore()
                    .AddControllerActivation();
            });

            InitializeContainer();
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            else
            {
                app.UseHsts();
            }

            app.UseHttpsRedirection();

            app.UseRouting();

            app.UseCors(x => x
                .WithMethods("OPTIONS", "GET", "POST", "PUT", "DELETE", "PATCH")
                .AllowAnyOrigin()
                //.AllowCredentials()
                .AllowAnyHeader()
                .WithExposedHeaders("x-perpro", "Cache-control", "Pragma", "x-hubin-info")
            );

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapGet("/", async context =>
                {
                    await context.Response.WriteAsync("Hello World!");
                });

                endpoints.MapControllers();
            });

            if(__simpleInjectorVerificationInsideScope__ == true)
            {
                __injectorContainer__.Verify();
            }
        }

        private void InitializeContainer()
        {
            // Add application services. For instance:
            Bootstrap.Register(__injectorContainer__, typeof(Program));
        }
    }

To Reproduce

Just start my project

Exception message:

Unhandled exception. System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.ChangeTokenRegistration`1..ctor(Func`1 changeTokenProducer, Action`1 changeTokenConsumer, TState state)
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Wevo.IO.Agent.Flow.Program.Main(String[] args) in /src/src/Wevo.IO.Agent.Flow.Runtime.API/Program.cs

Further technical details

• ASP.NET Core version - 3.1
• Docker image mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim

Running Docker/Kubernetes on Ubuntu 18 / Alpine

I've seen the same error on other issues (#8449, #11038, #7531) but they all are on 2.1, currently im running version 3.1.

[analogrelay]

@Gprsk do you have any other containers running on the same node? Did you try any of the mitigations/etc. described in https://github.com/dotnet/aspnetcore/issues/7531 ? Yes, they are on 2.1, but I don't think there's been any change in 3.1 that would change our behavior. The inotify limits are per-user and configured on the host machine (i.e. the node) rather than the container, so if you have a significant number of containers running on the same node this problem could be exasperated.

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.