Exception from CancellationTokenSource in Finalizer of FileSystemWatcher taking down whole application

[videokojot]

Problem: We are running into unsolvable problem, our application is taken down by this exception (and thats everything we were able to log from it in UnhandledException event) and we do not know how to localize it. This is from our logs:

Unhandled exception AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.); InnerException: Object reference not set to an instance of an object. caught by CurrentDomain.UnhandledException. Stacktrace: at System.Threading.CancellationTokenSource.CallbackNode.<>c.<ExecuteCallback>b__10_0(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   --- End of inner exception stack trace ---
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   at System.IO.FileSystemWatcher.StopRaisingEvents()
   at System.IO.FileSystemWatcher.Dispose(Boolean disposing)
   at System.ComponentModel.Component.Finalize()

Details: Some (maybe) relevant information:

• multiple WebApi ASP.NET Core microservice projects
• we are using .NET Core 2.2
• we are running in Linux docker (base image mcr.microsoft.com/dotnet/core/aspnet:2.2) managed by kubernetes
• we are using Serilog
• we do not use FileSystemWatcher class in our code (or in any of the referenced Nugets)
• we do not use anything like reloadOnChange: true for appsetting.json
• This can be connected to: https://github.com/dotnet/corefx/issues/32024 @shanselman

Also if we run: kubectl logs ${pod} --previous we can see this:

...
   State:          Running 
   Started:      Thu, 24 Oct 2019 10:28:21 +0200
   Last State:     Terminated
   Reason:       Error
    Exit Code:    134
    Started:      Thu, 24 Oct 2019 10:01:53 +0200
    Finished:     Thu, 24 Oct 2019 10:28:20 +0200   
...

The result from: kubectl logs $(kubectl get pods --selector=app=dataconnector -o=name) --previous is the same as the logged exception except there is line Aborted (core dumped)

What we tried (nothing helped):

• run application with debugger attached and wait for exception to happen, unfortunately when it happens Visual Studio shows 'in external code' and we do not get anything better that...
• set DOTNET_USE_POLLING_FILE_WATCHER=true to our Dockerfile
  • https://github.com/dotnet/corefx/issues/32024
• we have question on SO: https://stackoverflow.com/questions/58444716/unhandled-exception-in-finalizer-not-from-our-code
• try to find any usage of FileSystemWatcher in our code (or nugets)
  • we decompiled whole project output (self contained publish) with ILSpy -> Saved as C# files -> tried to find any usage of FileSystemWatcher
  • we found only usgae in this file (which we do not use - as our project is WebApi) https://github.com/aspnet/JavaScriptServices/blob/master/src/Microsoft.AspNetCore.NodeServices/HostingModels/OutOfProcessNodeInstance.cs

I am not sure if it is belong here, but FileSystemWatcher.cs is from here, but it can be in coreclr (because of CancelationTokeSource.cs) or in apsnetcore...

Are you having any thoughts? Currently this is blocking us moving forward, as all of our microservice are dying unexpectedly...

[stephentoub]

we are using .NET Core 2.2

What exact version of the runtime? This looks very similar to https://github.com/dotnet/corefx/issues/33844, which was fixed in .NET Core 2.2.1.

[videokojot]

@stephentoub we are using

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-restore for build and FROM mcr.microsoft.com/dotnet/core/aspnet:2.2 AS runtime-image for runtime

it seem that they are currently using 2.2.402 (sdk) and 2.2.7 (runtime img) according their docker files: https://github.com/dotnet/dotnet-docker/blob/master/2.2/sdk/stretch/amd64/Dockerfile and https://github.com/dotnet/dotnet-docker/blob/master/2.2/aspnet/stretch-slim/amd64/Dockerfile

We have run docker prune only few days ago, so it is not possible, that there would be older version of 2.2. cached...

Thank you for looking into this.