search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.08k stars 4.7k forks source link

System.IO.IOException : The configured user limit (1024) on the number of inotify instances has been reached. #40350

Closed mike-jewell closed 4 years ago

mike-jewell commented 4 years ago

We have an Azure App Service which has 5 containers running. 3 are web apps, 2 are function apps. Things have been running fine, but we have recently upgraded the nuget and dotnet core containers used on 1 web app and 1 function app. Since then we have started to see this error.

As this is spread over 5 apps and also 5 different solutions I have no idea where to start on a repo to demonstrate this. The functions app has 35 functions mixed between queues, timers and container watches. I cannot configure the json watchwithin the function app as far as I am aware, and I haven't, but I am getting the following from the function app:

2020-08-04T17:48:26.658964184Z Starting OpenBSD Secure Shell server: sshd.
2020-08-04T17:48:27.107082516Z Unhandled exception. System.IO.IOException: The configured user limit (1024) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
2020-08-04T17:48:27.107125317Z    at System.IO.FileSystemWatcher.StartRaisingEvents()
2020-08-04T17:48:27.107136818Z    at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
2020-08-04T17:48:27.107146618Z    at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
2020-08-04T17:48:27.107157318Z    at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
2020-08-04T17:48:27.107167318Z    at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
2020-08-04T17:48:27.107176218Z    at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
2020-08-04T17:48:27.107185019Z    at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
2020-08-04T17:48:27.107194719Z    at Microsoft.Extensions.Primitives.ChangeToken.ChangeTokenRegistration`1..ctor(Func`1 changeTokenProducer, Action`1 changeTokenConsumer, TState state)
2020-08-04T17:48:27.107203819Z    at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
2020-08-04T17:48:27.107281021Z    at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
2020-08-04T17:48:27.107297621Z    at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
2020-08-04T17:48:27.107307022Z    at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
2020-08-04T17:48:27.107315922Z    at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
2020-08-04T17:48:27.107325022Z    at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
2020-08-04T17:48:27.107333822Z    at Microsoft.Azure.WebJobs.Script.WebHost.Program.BuildWebHost(String[] args) in /src/azure-functions-host/src/WebJobs.Script.WebHost/Program.cs:line 35
2020-08-04T17:48:27.107343722Z    at Microsoft.Azure.WebJobs.Script.WebHost.Program.Main(String[] args) in /src/azure-functions-host/src/WebJobs.Script.WebHost/Program.cs:line 27
2020-08-04T17:48:27.429690099Z /azure-functions-host/start.sh: line 28:    18 Aborted                 (core dumped) /azure-functions-host/Microsoft.Azure.WebJobs.Script.WebHost

The web apps have been deliberately configured with:

.AddJsonFile("appsettings.json", optional: false, reloadOnChange: false)

As we have seen this before about 2 months ago. However I found that I had deployed a web app with AddRazorRuntimeCompilation(); on which was cxausing the issue.

The web app was also reporting the issue against the following stack trace:

System.IO.FileSystemWatcher.StartRaisingEvents():131
System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed():31
System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value):42

Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher():56
Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter):57
Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter):17
Microsoft.AspNetCore.Mvc.Razor.Infrastructure.DefaultFileVersionProvider.AddFileVersionToPath(PathString requestPathBase, String path):85
Microsoft.AspNetCore.Mvc.TagHelpers.ScriptTagHelper.Process(TagHelperContext context, TagHelperOutput output):196
Microsoft.AspNetCore.Razor.TagHelpers.TagHelper.ProcessAsync(TagHelperContext context, TagHelperOutput output):8
Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperRunner.RunAsync(TagHelperExecutionContext executionContext):100
AspNetCore.Views_Shared__Layout+<<ExecuteAsync>b__20_1>d.MoveNext() in /app/Bayleys.Processor.Web/Views/Shared/_Layout.cshtml:45

Nothing else within the applications hosted on the server utilise filewatchers outside of external dependencies. We upgraded the nuget packages, and have now used the following across the 2 apps:

Top-level Package                                                   Requested                Resolved
> AutoMapper                                                        10.0.0                   10.0.0
> AutoMapper.Extensions.Microsoft.DependencyInjection               8.0.1                    8.0.1
> AWSSDK.S3                                                         3.3.111.33               3.3.111.33
> Cronos                                                            0.7.0                    0.7.0
> Dapper                                                            2.0.35                   2.0.35
> Dapper.Contrib                                                    2.0.35                   2.0.35
> DapperExtensions.DotnetCore                                       1.0.1                    1.0.1
> DotLiquid                                                         2.0.361                  2.0.361
> DotNet.Glob                                                       3.0.9                    3.0.9
> FluentScheduler                                                   5.4.0                    5.4.0
> ElmahCore                                                         1.2.5                    1.2.5
> Google.Apis                                                       1.48.0                   1.48.0
> Google.Apis.Analytics.v3                                          1.48.0.1679              1.48.0.1679
> Google.Apis.Auth                                                  1.48.0                   1.48.0
> Jil                                                               3.0.0-alpha2             3.0.0-alpha2
> Magick.NET-Q8-AnyCPU                                              7.21.1                   7.21.1
> MailKit                                                           2.8.0                    2.8.0
> Microsoft.ApplicationInsights                                     2.14.0                   2.14.0
> Microsoft.ApplicationInsights.AspNetCore                          2.14.0                   2.14.0
> Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel      2.14.0                   2.14.0
> Microsoft.AspNetCore.Authorization                                3.1.6                    3.1.6
> Microsoft.AspNetCore.DataProtection.StackExchangeRedis            3.1.6                    3.1.6
> Microsoft.AspNetCore.Identity.EntityFrameworkCore                 3.1.6                    3.1.6
> Microsoft.AspNetCore.Mvc.NewtonsoftJson                           3.1.6                    3.1.6
> Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation                 3.1.6                    3.1.6
> Microsoft.AspNetCore.Authentication.AzureAD.UI                    3.1.6                    3.1.6
> Microsoft.Azure.Cosmos.Table                                      1.0.7                    1.0.7
> Microsoft.Azure.Storage.Blob                                      11.2.0                   11.2.0
> Microsoft.Azure.Storage.Queue                                     11.2.0                   11.2.0
> Microsoft.EntityFrameworkCore                                     3.1.6                    3.1.6
> Microsoft.EntityFrameworkCore.Design                              3.1.6                    3.1.6
> Microsoft.EntityFrameworkCore.InMemory                            3.1.6                    3.1.6
> Microsoft.EntityFrameworkCore.Relational                          3.1.6                    3.1.6
> Microsoft.EntityFrameworkCore.Sqlite                              3.1.6                    3.1.6
> Microsoft.EntityFrameworkCore.SqlServer                           3.1.6                    3.1.6
> Microsoft.EntityFrameworkCore.SqlServer.NetTopologySuite          3.1.6                    3.1.6
> Microsoft.EntityFrameworkCore.Tools                               3.1.6                    3.1.6
> Npgsql.EntityFrameworkCore.PostgreSQL                             3.1.4                    3.1.4
> Microsoft.Extensions.Caching.Abstractions                         3.1.6                    3.1.6
> Microsoft.Extensions.Configuration                                3.1.6                    3.1.6
> Microsoft.Extensions.Configuration.Abstractions                   3.1.6                    3.1.6
> Microsoft.Extensions.Configuration.Binder                         3.1.6                    3.1.6
> Microsoft.Extensions.Configuration.FileExtensions                 3.1.6                    3.1.6
> Microsoft.Extensions.Configuration.Json                           3.1.6                    3.1.6
> Microsoft.Extensions.DependencyInjection                          3.1.6                    3.1.6
> Microsoft.Extensions.DependencyInjection.Abstractions             3.1.6                    3.1.6
> Microsoft.Extensions.FileProviders.Abstractions                   3.1.6                    3.1.6
> Microsoft.Extensions.FileProviders.Embedded                       3.1.6                    3.1.6
> Microsoft.Extensions.FileProviders.Physical                       3.1.6                    3.1.6
> Microsoft.Extensions.Hosting                                      3.1.6                    3.1.6
> Microsoft.Extensions.Http                                         3.1.6                    3.1.6
> Microsoft.Extensions.Logging                                      3.1.6                    3.1.6
> Microsoft.Extensions.Logging.Abstractions                         3.1.6                    3.1.6
> Microsoft.Extensions.Logging.Console                              3.1.6                    3.1.6
> Microsoft.Extensions.ObjectPool                                   3.1.6                    3.1.6
> Microsoft.Extensions.PlatformAbstractions                         1.1.0                    1.1.0
> Microsoft.AspNetCore.Mvc.Core                                     2.2.5                    2.2.5
> Microsoft.Azure.Functions.Extensions                              1.0.0                    1.0.0
> Microsoft.Azure.WebJobs                                           3.0.16                   3.0.16
> Microsoft.Azure.WebJobs.Extensions.Storage                        3.0.10                   3.0.10
> Microsoft.NET.Sdk.Functions                                       3.0.7                    3.0.7
> Microsoft.SourceLink.AzureRepos.Git                               1.0.0                    1.0.0
> Microsoft.VisualStudio.Azure.Containers.Tools.Targets             1.10.8                   1.10.8
> Microsoft.VisualStudio.Web.CodeGeneration.Design                  3.1.3                    3.1.3
> Mindscape.Raygun4Net.NetCore                                      6.3.0                    6.3.0
> Mindscape.Raygun4Net.AspNetCore                                   6.3.0                    6.3.0
> NetTopologySuite                                                  2.0.0                    2.0.0
> NetTopologySuite.IO.SqlServerBytes                                2.0.0                    2.0.0
> Newtonsoft.Json                                                   12.0.3                   12.0.3
> Polly                                                             7.2.1                    7.2.1
> protobuf-net                                                      3.0.29                   3.0.29
> Serilog                                                           2.9.0                    2.9.0
> Serilog.AspNetCore                                                3.4.0                    3.4.0
> Serilog.Extensions.Logging                                        3.0.1                    3.0.1
> Serilog.Settings.Configuration                                    3.1.0                    3.1.0
> Serilog.Sinks.ApplicationInsights                                 3.1.0                    3.1.0
> Serilog.Sinks.Literate                                            3.0.0                    3.0.0
> Serilog.Sinks.RollingFile                                         3.3.0                    3.3.0
> Sendgrid                                                          9.19.0                   9.19.0
> SharpZipLib                                                       1.2.0                    1.2.0
> StackExchange.Redis                                               2.1.58                   2.1.58
> System.ComponentModel.Annotations                                 4.7.0                    4.7.0
> System.ComponentModel.TypeConverter                               4.3.0                    4.3.0
> System.Data.SqlClient                                             4.8.1                    4.8.1
> System.Diagnostics.TraceSource                                    4.3.0                    4.3.0
> System.Linq.Queryable                                             4.3.0                    4.3.0
> System.Net.Http                                                   4.3.4                    4.3.4
> System.Reflection.TypeExtensions                                  4.7.0                    4.7.0
> System.Text.Json                                                  4.7.2                    4.7.2
> TimeZoneConverter                                                 3.2.0                    3.2.0
> WindowsAzure.Storage                                              9.3.3                    9.3.3

Beyond what I've provided I have no idea what else I can do to demonstrate the problem.

Configuration

dotnet core 3.1 mcr.microsoft.com/dotnet/core/aspnet:3.1.6-buster-slim mcr.microsoft.com/azure-functions/dotnet:3.0 (just reverted from mcr.microsoft.com/azure-functions/dotnet:3.0-appservice which was also a change to this release)

All docker images use:

ENV DOTNET_USE_POLLING_FILE_WATCHER true

Not sure if that is the correct way of implementing this...

Azure App Service Linux Docker

Regression?

I think its a regression. Prior to the upgrade we were on the 3.1.4 versions of the main microsoft packages. and we were on the 3.1.4 docker image for the web app.

area-System.IO

Dotnet-GitSync-Bot commented 4 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

danmoseley commented 4 years ago

Related https://github.com/dotnet/runtime/issues/37664

fsalmeida commented 4 years ago

I've been facing the same problem as you do. Recently this problem has ocurred to one of my applications, then I've scaled up the ServicePlan and the application has back to life. Then I scaled it down and it still works. Today I've faced the same problem with other application and I'm still trying to find a way to solve the problem.

carlossanlop commented 4 years ago

I believe this is a duplicate of https://github.com/dotnet/runtime/issues/37664