Microsoft.Extensions.Configuration.Json leaves unobserved task exceptions in background tasks when using reloadOnChange:true

genriquez commented 3 weeks ago

Description

I've come across an error situations where background tasks are left with unhandled/unobserved task exceptions when using reloadOnChange: true while using filesystem watcher (instead of polling). This happens after the config file and its containing folder are deleted.

We use dispatcher/unhandled/unobserved exception handlers to do bug reporting and terminate the application in a safe manner whenever possible. Currently there is no way to hook into any kind of callback to handle these exceptions and prevent them from bubbling up in background.

Reproduction Steps

I've created a small repro case. You can just run them with dotnet run MSConfigCrashTests.csproj and some basic console logging while happen showing the state of the config loaded, and what handlers were triggered:

repro-delete-fswatcher.zip

Expected behavior

Background exceptions should not be left to bubble up as UnobservedTaskExceptions. If not by default, at least there should be a way to hook a handler and prevent the exception from bubbling up, like load exceptions can be handled (see attached project):

update-fswatcher.zip

Actual behavior

Background exceptions trigger the TaskScheduler.UnobservedTaskException handler with no other way of handling them. The exception is as follows:

System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (One or more errors occurred. (Error reading the C:\Users\XXX\AppData\Local\Temp\testdir\ directory.))
 ---> System.AggregateException: One or more errors occurred. (Error reading the C:\Users\XXX\AppData\Local\Temp\testdir\ directory.)
 ---> System.IO.FileNotFoundException: Error reading the C:\Users\XXX\AppData\Local\Temp\testdir\ directory.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.ChangeTokenRegistration`1.OnChangeTokenFired()
   at Microsoft.Extensions.Primitives.ChangeToken.ChangeTokenRegistration`1.<>c.<RegisterChangeTokenCallback>b__7_0(Object s)
   at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
   at System.Threading.CancellationTokenSource.Register(Delegate callback, Object stateForCallback, SynchronizationContext syncContext, ExecutionContext executionContext)
   at System.Threading.CancellationToken.Register(Delegate callback, Object state, Boolean useSynchronizationContext, Boolean useExecutionContext)
   at System.Threading.CancellationToken.UnsafeRegister(Action`1 callback, Object state)
   at Microsoft.Extensions.Internal.ChangeCallbackRegistrar.UnsafeRegisterChangeCallback[T](Action`1 callback, Object state, CancellationToken token, Action`1 onFailure, T onFailureState)
   at Microsoft.Extensions.Primitives.CancellationChangeToken.RegisterChangeCallback(Action`1 callback, Object state)
   at Microsoft.Extensions.Primitives.ChangeToken.ChangeTokenRegistration`1.RegisterChangeTokenCallback(IChangeToken token)
   at Microsoft.Extensions.Primitives.ChangeToken.ChangeTokenRegistration`1.OnChangeTokenFired()
   at Microsoft.Extensions.Primitives.ChangeToken.ChangeTokenRegistration`1.<>c.<RegisterChangeTokenCallback>b__7_0(Object s)
   at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   --- End of inner exception stack trace ---
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.<>c.<.cctor>b__43_0(Object state)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   --- End of inner exception stack trace ---

Regression?

No response

Known Workarounds

Two options:

Using polling with active polling seems to bypass this.
Wrapping the FileProvider so Watch(string) returns a wrapped IChangeToken that wraps actions passed to RegisterChangeCallback(Action<object?>,object?) to add a try/catch.

Configuration

$ dotnet --version
8.0.200

Repro project is net6. Still repros if changed to net8.

Other information

No response

dotnet-policy-service[bot] commented 3 weeks ago

Tagging subscribers to this area: @dotnet/area-extensions-configuration See info in area-owners.md if you want to be subscribed.

dotnet-policy-service[bot] commented 3 weeks ago

Tagging subscribers to this area: @dotnet/area-extensions-filesystem See info in area-owners.md if you want to be subscribed.

dotnet / runtime