search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.35k stars 4.74k forks source link

Random crash on Linux because assembly Microsoft.Win32.Registry cannot be loaded on aspnet 6.0.13 #81020

Open karlra opened 1 year ago

karlra commented 1 year ago

Is there an existing issue for this?

Describe the bug

At a random point, where according to the logs it appears nobody was using our web app, the app stopped accepting requests entirely. The app had an uptime of over a month at the point where this happened and nothing like this has happened before on any of our apps, which all have very long uptimes and most of them have tens or hundreds of thousands of users.

The following was spammed to the error logs, again and again (over 1000 such log entries over ~4 minutes until we manually restarted). It cannot be request driven because this is a low traffic app and does not have so many visitors in a 4 minute interval.

Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1] An unhandled exception has occurred while executing the request. System.Security.Cryptography.CryptographicException: An error occurred while trying to encrypt the provided data. Refer to the inner exception for more information. For more information go to http://aka.ms/dataprotectionwarning ---> System.TypeInitializationException: The type initializer for 'Microsoft.Extensions.Logging.LoggingExtensions' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Win32.Registry, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

File name: 'Microsoft.Win32.Registry, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' --- End of inner exception stack trace --- at Microsoft.Extensions.Logging.LoggingExtensions.ErrorOccurredWhileRefreshingKeyRing(ILogger logger, Exception exception) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext) --- End of inner exception stack trace --- at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext) at Microsoft.AspNetCore.Session.CookieProtection.Protect(IDataProtector protector, String data) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

Expected Behavior

The above does not happen. Why is it trying to load a Win32 registry module on Linux?

Steps To Reproduce

Unknown, happened randomly on a production system

Exceptions (if any)

Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1] An unhandled exception has occurred while executing the request. System.Security.Cryptography.CryptographicException: An error occurred while trying to encrypt the provided data. Refer to the inner exception for more information. For more information go to http://aka.ms/dataprotectionwarning ---> System.TypeInitializationException: The type initializer for 'Microsoft.Extensions.Logging.LoggingExtensions' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Win32.Registry, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

File name: 'Microsoft.Win32.Registry, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' --- End of inner exception stack trace --- at Microsoft.Extensions.Logging.LoggingExtensions.ErrorOccurredWhileRefreshingKeyRing(ILogger logger, Exception exception) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext) --- End of inner exception stack trace --- at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext) at Microsoft.AspNetCore.Session.CookieProtection.Protect(IDataProtector protector, String data) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

.NET Version

6.0.13

Anything else?

Running on Ubuntu 18.04.

global.json file: Not found

Host: Version: 6.0.13 Architecture: x64 Commit: 1af80ba017

.NET SDKs installed: No SDKs were found.

.NET runtimes installed: Microsoft.AspNetCore.App 6.0.13 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.13 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Download .NET: https://aka.ms/dotnet-download

Learn about .NET Runtimes and SDKs: https://aka.ms/dotnet/runtimes-sdk-info

dotnet-issue-labeler[bot] commented 1 year ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-microsoft-win32 See info in area-owners.md if you want to be subscribed.

Issue Details
### Is there an existing issue for this? - [X] I have searched the existing issues ### Describe the bug At a random point, where according to the logs it appears nobody was using our web app, the app stopped accepting requests entirely. The app had an uptime of over a month at the point where this happened and nothing like this has happened before on any of our apps, which all have very long uptimes and most of them have tens or hundreds of thousands of users. The following was spammed to the error logs, again and again (over 1000 such log entries over ~4 minutes until we manually restarted). It cannot be request driven because this is a low traffic app and does not have so many visitors in a 4 minute interval. Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1] An unhandled exception has occurred while executing the request. System.Security.Cryptography.CryptographicException: An error occurred while trying to encrypt the provided data. Refer to the inner exception for more information. For more information go to http://aka.ms/dataprotectionwarning ---> System.TypeInitializationException: The type initializer for 'Microsoft.Extensions.Logging.LoggingExtensions' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Win32.Registry, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified. File name: 'Microsoft.Win32.Registry, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' --- End of inner exception stack trace --- at Microsoft.Extensions.Logging.LoggingExtensions.ErrorOccurredWhileRefreshingKeyRing(ILogger logger, Exception exception) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext) --- End of inner exception stack trace --- at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext) at Microsoft.AspNetCore.Session.CookieProtection.Protect(IDataProtector protector, String data) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task) ### Expected Behavior The above does not happen. Why is it trying to load a Win32 registry module on Linux? ### Steps To Reproduce Unknown, happened randomly on a production system ### Exceptions (if any) Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1] An unhandled exception has occurred while executing the request. System.Security.Cryptography.CryptographicException: An error occurred while trying to encrypt the provided data. Refer to the inner exception for more information. For more information go to http://aka.ms/dataprotectionwarning ---> System.TypeInitializationException: The type initializer for 'Microsoft.Extensions.Logging.LoggingExtensions' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Win32.Registry, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified. File name: 'Microsoft.Win32.Registry, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' --- End of inner exception stack trace --- at Microsoft.Extensions.Logging.LoggingExtensions.ErrorOccurredWhileRefreshingKeyRing(ILogger logger, Exception exception) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext) --- End of inner exception stack trace --- at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext) at Microsoft.AspNetCore.Session.CookieProtection.Protect(IDataProtector protector, String data) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task) ### .NET Version 6.0.13 ### Anything else? Running on Ubuntu 18.04. global.json file: Not found Host: Version: 6.0.13 Architecture: x64 Commit: 1af80ba017 .NET SDKs installed: No SDKs were found. .NET runtimes installed: Microsoft.AspNetCore.App 6.0.13 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.13 [/usr/share/dotnet/shared/Microsoft.NETCore.App] Download .NET: https://aka.ms/dotnet-download Learn about .NET Runtimes and SDKs: https://aka.ms/dotnet/runtimes-sdk-info
Author: karlra
Assignees: -
Labels: `area-Microsoft.Win32`, `untriaged`
Milestone: -
ViktorHofer commented 1 year ago

The above does not happen. Why is it trying to load a Win32 registry module on Linux?

Microsoft.Win32.Registry is part of the shared framework and is used when resolving any of these types: https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.Win32.Registry/ref/Microsoft.Win32.Registry.cs. Do you use reflection or load assemblies dynamically, given that this happened randomly after a month?

Steps To Reproduce Unknown, happened randomly on a production system

Unfortunately we can't investigate this issue further without repro steps or a full stack trace.

ghost commented 1 year ago

This issue has been marked needs-author-action and may be missing some important information.

karlra commented 1 year ago

No, we are not using reflection or loading any assemblies dynamically. This is a very simple MVC app, I can't imagine that this is driven by user code. The stack trace I provided refers to internal Microsoft code, no?

ViktorHofer commented 1 year ago

@javiercn can we please move that issue back into aspnetcore? From the stacktrace, https://github.com/dotnet/aspnetcore/blob/73690575fd21834bda000f542c0d686427f113ae/src/DataProtection/DataProtection/src/KeyManagement/KeyRingProvider.cs#L206 throws the FileNotFoundException so something on that path must depend on Win32.Registry types.

karlra commented 1 year ago

Can I take a guess that the issue is here: https://github.com/dotnet/aspnetcore/blob/73690575fd21834bda000f542c0d686427f113ae/src/DataProtection/DataProtection/src/LoggingExtensions.cs#L182

on line 157-158, where it references RegistryKey and using Microsoft.Win32 in the header?

ViktorHofer commented 1 year ago

That makes sense yes. What I don't understand is why the FileNotFoundException happens given that Microsoft.Win32.Registry.dll is part of the targeting pack, the shared framework and the runtime pack on all platforms. Even though it does throw PlatformNotSupportedExceptions on Unix, it's still included in the layout.

cc @dougbu @wtgodbe @JamesNK (as this is an aspnetcore issue)

dougbu commented 1 year ago

as this is an aspnetcore issue

That's not clear to me because the OP describes the issue as intermittent and happening in an app that's been running smoothly for weeks before the problem is hit. It's as if the file is getting deleted randomly.

If the problem occurs around the same time as a Visual Studio or Windows Update on the affected machine(s), that might be the explanation. I don't have a better idea at this point.

karlra commented 1 year ago

I just checked on the production server (that experienced this issue) and the file is there, in the /refs folder. It has correct permissions. Very strange.

aaron-puhl commented 1 year ago

Had the same issue on linux with .net7, but after redeploy it went away and I can't reproduce it.

tspence commented 11 months ago

I have observed this issue as well on DotNet 7.

System.Security.Cryptography.CryptographicException: An error occurred while trying to encrypt the provided data. Refer to the inner exception for more information. For more information go to http://aka.ms/dataprotectionwarning
 ---> System.TypeInitializationException: The type initializer for 'Microsoft.Extensions.Logging.LoggingExtensions' threw an exception.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Win32.Registry, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

The symptom is:

Since this issue only appears to happen after time, my hunch is that some logging-related object in this application was disposed. The best guess I have is that there is some tiny difference between the initial creation of the logging related object and the attempted recreation of the logging related object, which causes the recreation to fail.

Will share more details if I discover anything from our traces.

tspence commented 11 months ago

Possibly related: https://github.com/Azure/azure-storage-net/issues/969

sfiruch commented 9 months ago

Stumbled into the same problem:

codymullins commented 8 months ago

This happened to us today. .NET 8 Blazor app on Ubuntu. Other services unaffected. Rebooting the VM was required to resolve the issue.

System.Security.Cryptography.CryptographicException: An error occurred while trying to encrypt the provided data. Refer to the inner exception for more information.
 ---> System.TypeInitializationException: The type initializer for 'Microsoft.Extensions.Logging.LoggingExtensions' threw an exception.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Win32.Registry, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

File name: 'Microsoft.Win32.Registry, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   --- End of inner exception stack trace ---
   at Microsoft.Extensions.Logging.LoggingExtensions.ErrorOccurredWhileRefreshingKeyRing(ILogger logger, Exception exception)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
   at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler.WriteNonceCookie(String nonce)
   at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler.HandleChallengeAsyncInternal(AuthenticationProperties properties)
   at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler.HandleChallengeAsync(AuthenticationProperties properties)
   at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.ChallengeAsync(AuthenticationProperties properties)
   at Microsoft.AspNetCore.Authentication.AuthenticationService.ChallengeAsync(HttpContext context, String scheme, AuthenticationProperties properties)
   at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.<>c__DisplayClass0_0.<<HandleAsync>g__Handle|0>d.MoveNext()
--- End of stack trace from previous location ---
   at PureBlazor.BlazorFrameworkAuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) in G:\repos\pureblazor\cms\src\PureBlazor\BlazorFrameworkAuthorizationMiddlewareResultHandler.cs:line 24
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.HandleException(HttpContext context, ExceptionDispatchInfo edi)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
huancz commented 7 months ago

tl;dr - combination of lazy dll loading and some linux-specific magic is probably the culprit. Workaround at the end.

Happens to our app too. Still running on .NET 6.0 and restarting the app helps. Usually it takes few months of uptime before it manifests. My first attempt to make sense of the exception stack: cookie auth (or any other code that uses DataProtector) is using existing valid keyring. At some point it expires and auto refresh tries to renew it. And as part of renewal it tries to log something, so static initializer of Microsoft.Extensions.Logging.LoggingExtensions runs - and fails with the above exception. The intended log message is lost, instead we get the message about bad or missing Microsoft.Win32.Registry.

But if Win32.Registry was just broken and never load on linux, that would require that nothing ever uses the LoggingExtensions. Not even generating new keyring - or the app would break on startup. And message about new keyring definitely IS in the log.

But... it's long running app, running on linux. Which allows file overwrite while they are in use by running app, without forcing restart. And .NET runtime upgrade is just one apt-get upgrade away. SO, it MAY be looking for exact patch version that was valid when the app was started, but no longer exists due to system upgrade (even when the message says it's looking for X.0.0.0).

With this in mind, I did some experiments, command ls -l /proc/<pid>/fd | grep in32.

First run after encountering the problem and restarting service:

lr-x------ 1 wwwdata nogroup 64 Apr  3 12:23 209 -> /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.28/Microsoft.Win32.Registry.dll

After another restart: (empty, dll is not loaded)

Testing instance of the same app that has been running for a long time without user interaction - no Win32.Registry, but...

lr-x------ 1 wwwdata nogroup 64 Apr  3 12:24 170 -> /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.20/Microsoft.Win32.Primitives.dll (deleted)

(note the .20 version instead of .28, and (deleted) at the end).

My conclusion (and repro... kind of):

So, workaround should be something like this in Program.cs:

public static async Task Main(string[] args)
{
    {
        // https://github.com/dotnet/runtime/issues/81020
        _ = typeof(Microsoft.Win32.Registry).Assembly;
    }
    ...
}

I'd rather load Microsoft.Extensions.Logging.LoggingExtensions, but it's internal class. I've added it to our app as a test, but only thing I can report now is that it doesn't fail at startup and the win32 .dll is always loaded in memory. Reporting back if it helped will probably take a long time.

AndyCoder123 commented 5 months ago

I'm getting the same issue with a .NET 8 RazorPages app on Ubuntu 22. Interestingly it happened in two different ways on two different servers in the cluster. In both cases, rebooting the servers resolved the issue.

The second one involves a different binary (Microsoft.AspNetCore.Diagnostics.Abstractions) but every other symptom was the same, i.e. the server was running fine for months before the issue happened, then the logs were flooded with 2Gb per day of errors.

The stack traces are below:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HN2FQ2B985GH", Request id "0HN2FQ2B985GH:00000001": An unhandled exception was thrown by the application.
      System.InvalidOperationException: The exception handler configured on ExceptionHandlerOptions produced a 404 status response. This InvalidOperationException containing the original exception was thrown since this is often due to a misconfigured ExceptionHandlingPath. If the exception handler is expected to return 404 status responses then set AllowStatusCode404Response to true.
       ---> System.Security.Cryptography.CryptographicException: An error occurred while trying to encrypt the provided data. Refer to the inner exception for more information.
       ---> System.TypeInitializationException: The type initializer for 'Microsoft.Extensions.Logging.LoggingExtensions' threw an exception.
       ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Win32.Registry, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

      File name: 'Microsoft.Win32.Registry, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
         --- End of inner exception stack trace ---
         at Microsoft.Extensions.Logging.LoggingExtensions.ErrorOccurredWhileRefreshingKeyRing(ILogger logger, Exception exception)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
         --- End of inner exception stack trace ---
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
         at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenSerializer.Serialize(AntiforgeryToken token)
         at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.Serialize(IAntiforgeryFeature antiforgeryFeature)
         at Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.GenerateForm(String actionName, String controllerName, Object routeValues, FormMethod method, Nullable`1 antiforgery, Object htmlAttributes)
         at UGIPCore.Pages.Pages_SecurityLogin.ExecuteAsync()
         at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
         at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
         at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
         at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
         at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
         --- End of inner exception stack trace ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.HandleException(HttpContext context, ExceptionDispatchInfo edi)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HN2FQ2DUKVBS", Request id "0HN2FQ2DUKVBS:00000001": An unhandled exception was thrown by the application.
      System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Diagnostics.Abstractions, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

      File name: 'Microsoft.AspNetCore.Diagnostics.Abstractions, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.HandleException(HttpContext context, ExceptionDispatchInfo edi)
         at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.HandleException(HttpContext context, ExceptionDispatchInfo edi)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
dgoldm commented 3 months ago

@huancz - it's been 4 months since you proposed your workaround. Can you please report the results?

AndyCoder123 commented 3 months ago

@huancz - it's been 4 months since you proposed your workaround. Can you please report the results?

I've had his workaround on my servers since June, under similar conditions, and haven't seen the issue again. So it's not a guarantee of a fix but seems to have helped..

Safirion commented 2 months ago

I have the same random crash (after long running) on Ubuntu with a web api in .net 8.

Could not load file or assembly 'System.Web.HttpUtility, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.\n\n

Is there any fix planed because actually this is ruining my app stability on time ?

The workaround is really painful as I have to figure out all the dll referenced and call it on main func... On big application it's really boring.

huancz commented 2 months ago

@Safirion

The issue seem to be architectural, and impossible to be "fixed". Proper "fix" is to make sure you restart your apps when you update dotnet runtime on your server. Or don't update the runtime when you can't afford to restart apps.

We have since moved most of our apps to podman containers where it can't happen (not because of this issue, it's just better for security, and there is no support for older runtimes on newest ubuntu).

edit: I guess they could implement some opt-in eager dll loading. Which, honestly, seems like an overkill to fix something that's essentially user error - broken server management policy. Maybe if broader use case could be found... I wouldn't hold my breath.

Safirion commented 2 months ago

An option to disable this lazy loading of dll or just use the later one instead of the one present at the moment of the application starts certainly fix this issue.

It's not a broken management policy to run daily update task and I don't think that it was to the developer to takes in charge the bad management of dll loading on Linux (windows .net runtime update don't crash .net apps currently running on it)

huancz commented 2 months ago

It's a bad policy, because you may think your apps are secure and up to date, but they are still using the old vulnerable dlls. Allowing them to use mixed dlls from multiple runtimes seems like recipe for huge trouble too.

Windows dotnet update requires you to restart windows if any app is using those dlls. Linux is much more forgiving.

Safirion commented 2 months ago

It's not ideal, I agree, but it's far better than a crash in production in both cases.

Edit : I will use this method in main to ensure all assemblies are loaded at application startup. It will delay my application start by few seconds but at least it should not crash after start. 🤞

public static void LoadAllAssemblies()
{
    Assembly rootAssembly = Assembly.GetEntryAssembly();
    HashSet<string> visited = new();
    Queue<Assembly> queue = new();

    queue.Enqueue(rootAssembly);

    while (queue.Any())
    {
        Assembly assembly = queue.Dequeue();
        visited.Add(assembly.FullName);

        AssemblyName[] references = assembly.GetReferencedAssemblies();
        foreach (AssemblyName reference in references)
        {
            if (!visited.Contains(reference.FullName))
                queue.Enqueue(Assembly.Load(reference));
        }
    }
}

@huancz Thanks for the idea 👍