Closed jaydeept closed 5 years ago
Eilon
commented
5 years ago Hi @jaydeept could you show what your code looks like? Also, would it be possible to include a better-formatted log output? It's quite difficult to read that log with all the wrapping in odd places.
Eilon
commented
5 years ago BTW this error presumably happens in any instance where there are a lot of file watchers. Configuration file watchers are a common case, but there can be any number of other cases.
You can also look at raising the limits on your system by doing something like what's described in the answers here: https://stackoverflow.com/questions/32281277/too-many-open-files-failed-to-initialize-inotify-the-user-limit-on-the-total
jaydeept
commented
5 years ago Sure thing @Eilon ,
Here is the stacktrace and formatted log:
----Autofac.Core.DependencyResolutionException : An error occurred during the activation of a particular registration.
See the inner exception for details.
Registration: Activator = DefaultActionDescriptorCollectionProvider (ReflectionActivator),
Services = [Microsoft.AspNetCore.Mvc.Infrastructure.IActionDescriptorCollectionProvider],
Lifetime = Autofac.Core.Lifetime.RootScopeLifetime,
Sharing = Shared,
Ownership = OwnedByLifetimeScope
---An exception was thrown while invoking the constructor 'Void.ctor(System.Collections.Generic.IEnumerable`1[Microsoft.AspNetCore.Mvc.Abstractions.IActionDescriptorProvider], System.Collections.Generic.IEnumerable`1[Microsoft.AspNetCore.Mvc.Infrastructure.IActionDescriptorChangeProvider])'
on type 'DefaultActionDescriptorCollectionProvider'.
---**The configured user limit (128) on the number of inotify instances has been reached. (See inner exception for details.)** (See inner exception for details.)
----Autofac.Core.DependencyResolutionException : An exception was thrown while invoking the constructor
'Void .ctor(System.Collections.Generic.IEnumerable`1[Microsoft.AspNetCore.Mvc.Abstractions.IActionDescriptorProvider], System.Collections.Generic.IEnumerable`1[Microsoft.AspNetCore.Mvc.Infrastructure.IActionDescriptorChangeProvider])'
on type 'DefaultActionDescriptorCollectionProvider'.
---The configured user limit (128) on the number of inotify instances has been reached. (See inner exception for details.)
----System.IO.IOException : The configured user limit (128) on the number of inotify instances has been reached.
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters)
at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
at Microsoft.AspNetCore.Builder.MvcApplicationBuilderExtensions.UseMvc(IApplicationBuilder app, Action`1 configureRoutes)
at WebApp.Startup.Configure(IApplicationBuilder app, IHostingEnvironment env) in .....\App_Start\Startup.cs:line 47
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.Configure(IApplicationBuilder app)
at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
at Microsoft.AspNetCore.Hosting.Internal.WebHost.StartAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.TestHost.TestServer..ctor(IWebHostBuilder builder, IFeatureCollection featureCollection)
**at WebAppTests.DependencyInjectionTests.OneTimeSetUp() in ....source\WebAppTests\DITests.cs:line 85**
--DependencyResolutionException
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
--DependencyResolutionException
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
--DependencyResolutionException
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
at System.Linq.Enumerable.SelectIPartitionIterator`2.PreallocatingToArray(Int32 count)
at System.Linq.Enumerable.SelectIPartitionIterator`2.ToArray()
at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
at Autofac.Features.Collections.CollectionRegistrationSource.<>c__DisplayClass0_0.<RegistrationsFor>b__0(IComponentContext c, IEnumerable`1 p)
at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
--DependencyResolutionException
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
--DependencyResolutionException
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
--DependencyResolutionException
at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
--IOException
at System.IO.FileSystemWatcher.StartRaisingEvents()
at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
at Microsoft.AspNetCore.Mvc.RazorPages.Internal.PageActionDescriptorChangeProvider.GetChangeToken()
at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
at Microsoft.AspNetCore.Mvc.Infrastructure.DefaultActionDescriptorCollectionProvider..ctor(IEnumerable`1 actionDescriptorProviders, IEnumerable`1 actionDescriptorChangeProviders)
at lambda_method(Closure , Object[] )
at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()And here is my code from Startup:
public class Startup
{
public static void Main(string[] args)
{
var configurationSettings = new ConfigurationSettings(args);
var properties = GetConfigurationProperties(configurationSettings);
var host = new WebHostBuilder()
.UseContentRoot(Directory.GetCurrentDirectory())
.UseStartup<Startup>()
.UseKestrel()
.UseUrls(configurationSettings.GetBaseAddress())
.ConfigureServices(services => services.AddProperties(properties))
.Build();
host.Run();
}
// This method gets called by the runtime. Use this method to add services to the container.
// For more information on how to configure your application, visit https://go.microsoft.com/fwlink/?LinkID=398940
public IServiceProvider ConfigureServices(IServiceCollection services)
{
services.AddAutofac();
services.AddLogging();
services.AddHttpContextAccessor();
services.AddMvc().AddSerializerSettings();
services.AddSwaggerGen(SwaggerConfig.ConfigureHelpDocuments);
return services.GetAutofacServiceProvider();
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
app.UseOptions();
app.UseRewriter(SetAliasesForDocumentationPage("doc", "docs", "help"));
app.UseStaticFiles();
app.UseMvc();
app.UseSwagger();
app.UseSwaggerUI(SwaggerConfig.ConfigureHelpUi);
}
private static RewriteOptions SetAliasesForDocumentationPage(params string[] aliasesForDoc)
{
var options = new RewriteOptions();
foreach (var aliasForDoc in aliasesForDoc)
{
options.AddRedirect(aliasForDoc, "swagger");
}
return options;
}And starcktrace has reported line DITests.cs:line 85 which is below one where I am using TestServer for my testing purpose. :
m_TestServer = new TestServer(webHostBuilder);Also regarding your post related to file watchers, I don't think I am using one. This is happening inside container as part of CI so I don't have privilege to modify the limits.conf file by going there.
jaydeept
commented
5 years ago Hello @Eilon , Any input on this issue? I wanted to confirm if it's my code which is opening so many files (though I don't think that's happening)
Eilon
commented
5 years ago @jaydeept sorry I'm not super sure. Maybe try disabling some of the features you're using and see if it makes a difference? There are some components in use there that might involve file watchers (though I'm not certain). Stuff such as Logging, Swagger, or other components. This would be helpful to at least see if we can figure out which specific component (if any) is causing this, and then figure out where to go from there.
HaoK
commented
5 years ago We are also seeing this in some helix queues (Fedora/some Ubuntu) now that identity functional tests are working, see https://mc.dot.net/#/user/aspnetcore/pr~2Faspnet~2Faspnetcore/ci/20190321.8/workItem/Microsoft.AspNetCore.Identity.FunctionalTests-netcoreapp3.0/wilogs
jacob-l
commented
5 years ago Hi, @Eilon Try to set up environment variable DOTNET_USE_POLLING_FILE_WATCHER=1
rtrimurthulu
commented
5 years ago https://github.com/Legedric/ptmagic/issues/165
follow this for ubuntu
I found the following blog post that might help pinpoint the problem and solution. Hope it helps.
https://www.danieldevelops.com/inotify-watcher-dotnet-core
Alternative is to increase the inotify limit, which I have done until you (hopefully) fix the issue.
Just for reference for others. You can get your current inotify file watch limit by executing:
$ cat /proc/sys/fs/inotify/max_user_watches
You can set a temporary new limit with:
$ sudo sysctl -w fs.inotify.max_user_watches=16384
If you like to make your limit permanent use:
$ echo fs.inotify.max_user_watches=16384 | sudo tee -a /etc/sysctl.conf $ sudo sysctl -p
Thx for the great work. Love the PT Magic!
After fix restart your vs code
rmam-spinworks
commented
5 years ago I can confirm this issue in Ubuntu 18.04. Running a brand new webapi project is enough to trigger the exception.
Steps to reproduce:
Xarkam
commented
5 years ago Same issue. The problem appeared in Jetbrain Rider while vs code is opened. When vscode it's closed, works well.
vscode 1.31.1 dotnet 2.2.203 OS: kde neon 18.04
I installed my system on a brand new drive a week ago and made the change to /proc/sys/fs/inotify/max_user_watches as requested by vscode.
By the way the value in my file is 524288
It's a value well beyond what @rtrimurthulu says in post. But I have this mistake.
mathavanmani
commented
5 years ago Hi @jaydeept , Have you found the root cause of this issue by any chance?. I'm also facing the same issue
rtrimurthulu
commented
5 years ago use command export DOTNET_USE_POLLING_FILE_WATCHER=true And restart VsCode it will work
ajamrozek
commented
5 years ago I get this with increasing consistency when deploying new Linux hosts to my on-prem enterprise k8s cluster.
I've assigned the configs/jsons to not refresh, supposedly disabling the file watch.
config.AddJsonFile("appsettings.json",true, false);
I've set an env var in my dockerfile
ENV DOTNET_USE_POLLING_FILE_WATCHER true
And I still usually, not always, get a fatal startup failing my pods. It goes away with an indeterminate number of pod refreshes. Locally the containers seem to startup consistently in my docker desktop k8s.
Unhandled Exception: System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached. at System.IO.FileSystemWatcher.StartRaisingEvents() at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed() at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value) at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher() at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter) at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter) at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__0_0() at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func1 changeTokenProducer, Action changeTokenConsumer)
at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()`
analogrelay
commented
5 years ago Can anyone provide a runnable sample project that reproduces this issue? That would help us diagnose this. It looks like something in config is still registering file system watchers. We do also have the appsettings.[environmentName].json file, which may be listed for refresh even if you disable the appsettings.json file refresh. Using the environment variable should be disabling inotify-based watching though.
ajamrozek
commented
5 years ago I've created a boilerplate ASP.Net Core 2.2 WebAPI project, with no additional functionality and been able to "reproduce" this error in our environment. Unfortunately, it's an intermittent issue. We can't track down what resources aren't available during startup (Linux host has 100's of free GB and 2GB mem cap on pods). So, while it's consistent enough for it to become a critical issue, it's not consistent enough for us to nail down in a 100% repro project.
analogrelay
commented
5 years ago We can't track down what resources aren't available during startup
My understanding is that there is a hard limit on the number of file watchers that you can have (the "inotify instances"), so even though you have a lot of free memory you have to actually adjust that limit to avoid this issue. You can do that with this command:
sudo sysctl fs.inotify.max_user_watches=<some new limit higher than 128>There are some issues with inotify and docker. How many containers do you have running per machine? The inotify instance limit is per user not per process so if you have a lot of processes running it can cause issues.
ajamrozek
commented
5 years ago It seems like our usage of the same user across all our pods in the cluster was creating this issue. I rewrote our dockerfiles to have their own unique users and our pods appear to be behaving better.
analogrelay
commented
5 years ago Ah yeah, that could definitely cause a problem like this. In that situation (where you intentionally have a lot of processes wiht the same user) it's also reasonable to increase this limit (as long as you know why the current limit isn't sufficient, which in this case you do).
Closing this for now on the supposition that this is due to sharing a single user for multiple applications. Please feel free to comment further, open a new issue, etc. if you're seeing this and don't fall into that category.
paillave
commented
5 years ago your linux docker must have this extra setup:
echo fs.inotify.max_user_instances=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
this will increase the amount of authorized instances as too many may be taken by something else
I am using .Net Core 2.2 for my web project. I got test cases which I am executing using
dotnet vstestas part of CI pipeline in Linux docker. However the execution of test cases fails sometimes with error "The configured user limit (128) on the number of inotify instances has been reached." However if I re-run, it works successfully. I am not sure what's going wrong. I had a look at this post but I am not usingJSONfile in myStartupclass, also I am callingbuild()only once in myMainmethod.Error Message: