search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.12k stars 4.7k forks source link

IOException inside HostBuilder #91879

Open chrfin opened 1 year ago

chrfin commented 1 year ago

Description

We use Host.CreateDefaultBuilder(args).RunConsoleAsync() to create a host and are hosting some custom services with that host.

Reproduction Steps

Sadly we cannot reproduce the problem deliberately, but it happens a few times per week...

Expected behavior

We would like the program to start successfully.

Actual behavior

From time to time this program crashes directly at startup with the following exception:

Unhandled exception. System.IO.IOException: Invalid argument : '/proc/1889/task/1889/net'
   at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance..ctor(FileSystemWatcher watcher, SafeFileHandle inotifyHandle, String directoryPath, Boolean includeSubdirectories, NotifyFilters notifyFilters, CancellationToken cancellationToken)
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.RunConsoleAsync(IHostBuilder hostBuilder, CancellationToken cancellationToken)

Regression?

No response

Known Workarounds

We have the feeling that it is somehow related to the ethernet configuration of our device as it seems to be not happening if we disconnect the ethernet cable.

Configuration

The program is running with .NET 6.0 on an embedded ARM64 based system running a custom Linux build based on kernel 5.4.

Other information

No response

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-system-io See info in area-owners.md if you want to be subscribed.

Issue Details
### Description We use `Host.CreateDefaultBuilder(args).RunConsoleAsync()` to create a host and are hosting some custom services with that host. ### Reproduction Steps Sadly we cannot reproduce the problem deliberately, but it happens a few times per week... ### Expected behavior We would like the program to start successfully. ### Actual behavior From time to time this program crashes directly at startup with the following exception: ``` Unhandled exception. System.IO.IOException: Invalid argument : '/proc/1889/task/1889/net' at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext() at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName) at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName) at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName) at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName) at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName) at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName) at System.IO.FileSystemWatcher.RunningInstance..ctor(FileSystemWatcher watcher, SafeFileHandle inotifyHandle, String directoryPath, Boolean includeSubdirectories, NotifyFilters notifyFilters, CancellationToken cancellationToken) at System.IO.FileSystemWatcher.StartRaisingEvents() at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed() at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value) at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher() at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter) at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter) at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0() at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer) at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source) at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration() at Microsoft.Extensions.Hosting.HostBuilder.Build() at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.RunConsoleAsync(IHostBuilder hostBuilder, CancellationToken cancellationToken) ``` ### Regression? _No response_ ### Known Workarounds We have the feeling that it is somehow related to the ethernet configuration of our device as it seems to be not happening if we disconnect the ethernet cable. ### Configuration The program is running with .NET 6.0 on an embedded ARM64 based system running a custom Linux build based on kernel 5.4. ### Other information _No response_
Author: chrfin
Assignees: -
Labels: `area-System.IO`
Milestone: -
jozkee commented 1 year ago

@chrfin would you mind sharing a minimal repro?

chrfin commented 1 year ago

@Jozkee Sadly I cannot, as we cannot even reproduce it ourselves, but it happens a few times a week when starting the app several times a day.

But as mentioned above we use the Host.CreateDefaultBuilder(args) without any special configuration or even a JSON config file. But the host-builder seems to try to read some config-file as far as I understand the stack-trace.

One additional information I missed: This happens directly after booting, as the program auto-starts, and the ethernet-connection is initialized at the same time.

I hoped someone with insights into this area has an idea how such an exception could pop up and crash the application.
I guess there is some kind of unhandled race-condition and it tries to create a file-watcher for a file no longer existing a few moments later...

Because of the "net" in the error path and the fact it does not happen with disconnected ethernet we guessed it must be some file used during configuring the ethernet interface.

Here the part of our startup-logic causing the problem:

  await Host.CreateDefaultBuilder(args)
     .ConfigureLogging(logging =>
      {
        logging.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Warning);
      })
     .ConfigureServices((_, services) =>
      {
        // add and start Watchdog service
        if (!File.Exists(CommonDefinitions.NoWatchdogFile))
            services.AddHostedService<WatchdogService>();
        // Start Factory Reset Handler Service as a controlled service
        if (!File.Exists(CommonDefinitions.NoFactoryResetHandlerFile))
            services.AddHostedService<FactoryResetHandlerService>();
    }).RunConsoleAsync().ConfigureAwait(false);
jozkee commented 1 year ago

I'm also curious about the path of your config.json (part of the repro). I don't understand why the FSW should be iterating over /proc.

chrfin commented 1 year ago

We do not have any config.json file at all. And args contians only "-w" (we use that some other place).

inigoOCF commented 1 year ago

Same behaviour. We have been working for a week with our application installed on about 8 computers (Ubuntu 20.04 and 22.04) and today 3 of them have launched the exception at start up:

sep 14 09:02:17   Unhandled exception. System.IO.IOException: Invalid argument : '/proc/1881/task/1881/net'
sep 14 09:02:17      at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
sep 14 09:02:17      at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
sep 14 09:02:17      at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
sep 14 09:02:17      at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
sep 14 09:02:17      at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
sep 14 09:02:17      at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
sep 14 09:02:17      at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
sep 14 09:02:17      at System.IO.FileSystemWatcher.RunningInstance..ctor(FileSystemWatcher watcher, SafeFileHandle inotifyHandle, String directoryPath, Boolean includeSubdirectories, NotifyFilters notifyFilters, CancellationToken cancellationToken)
sep 14 09:02:17      at System.IO.FileSystemWatcher.StartRaisingEvents()
sep 14 09:02:17      at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
sep 14 09:02:17      at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
sep 14 09:02:17      at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
sep 14 09:02:17      at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
sep 14 09:02:17      at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
sep 14 09:02:17      at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
sep 14 09:02:17      at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
sep 14 09:02:17      at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
sep 14 09:02:17      at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
sep 14 09:02:17      at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
sep 14 09:02:17      at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
sep 14 09:02:17      at Microsoft.Extensions.Configuration.ConfigurationExtensions.Add[TSource](IConfigurationBuilder builder, Action`1 configureSource)
sep 14 09:02:17      at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(IConfigurationBuilder builder, IFileProvider provider, String path, Boolean optional, Boolean reloadOnChange)
sep 14 09:02:17      at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(IConfigurationBuilder builder, String path, Boolean optional, Boolean reloadOnChange)
sep 14 09:02:17      at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.<>c__DisplayClass11_0.<ConfigureDefaults>b__1(HostBuilderContext hostingContext, IConfigurationBuilder config)
sep 14 09:02:17      at Microsoft.AspNetCore.Hosting.BootstrapHostBuilder.RunDefaultCallbacks(ConfigurationManager configuration, HostBuilder innerBuilder)
sep 14 09:02:17      at Microsoft.AspNetCore.Builder.WebApplicationBuilder..ctor(WebApplicationOptions options, Action`1 configureDefaults)
sep 14 09:02:17      at Microsoft.AspNetCore.Builder.WebApplication.CreateBuilder()

startup-logic:

                IConfiguration configuration = new ConfigurationManager();
                configuration["AllowedHosts"] = "*";
                configuration["Kestrel:EndpointDefaults:Protocols"] = "Http2";
                configuration["Logging:LogLevel:Default"] = "Information";
                configuration["Logging:LogLevel:Microsoft.AspNetCore"] = "Warning";

                var serverUrl = $"http://localhost:{port}";

                var builder = WebApplication.CreateBuilder();
                builder.Configuration.AddConfiguration(configuration);
                builder.Services.AddGrpc();
                builder.Services.AddSingleton<IServiceGrpcServer>(this);
                _webApplication = builder.Build();
                _webApplication.MapGrpcService<ServiceGrpcServer>();
                _webApplication.RunAsync(serverUrl);
steveadvanced commented 11 months ago

We're seeing the same issue - starting our dotnet application on boot on Linux gives this same error every now and then. The application will run fine if started up manually afterwards, so it does appear to be a timing issue when the system boots up. Any fix or workaround would be appreciated!

Unhandled exception. System.IO.IOException: Invalid argument : '/proc/319/task/319/net'
   at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance..ctor(FileSystemWatcher watcher, SafeFileHandle inotifyHandle, String directoryPath, Boolean includeSubdirectories, NotifyFilters notifyFilters, CancellationToken cancellationToken)
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at App.Program.Main(String[] args)
   at App.Program.<Main>(String[] args)
chrfin commented 11 months ago

@Jozkee Any chance for a feasible work-around or to get a fix into .NET 8.0 (as 8 is LTS and 9 will not)?

jozkee commented 11 months ago

As a workaround, can you try setting env. var. DOTNET_USE_POLLING_FILE_WATCHER=1? Since this is an issue with how the FileSystemWatcher traverses the directory, is possible that the polling doesn't incur in enumerating /proc subdirectories.

Could you please share in which directory your app is on disk? I'm wondering if it's on / and that's why /proc is being enumerated for watching.

chrfin commented 11 months ago

@Jozkee Our app is at /usr/USER/APP/app. It is started with a specific user, maybe that changes the working dir?

sancheolz commented 4 days ago

/proc is filesystem for process states. in /proc/{id} the id is process id. For me state of process with id mentioned in /proc was zombie. So to reproduce issue you need to put some process to zombie state, and than start your service with hostbuilder.