search

dotnet / sdk

Core functionality needed to create .NET Core projects, that is shared between Visual Studio and CLI
https://dot.net/core
MIT License
2.75k stars 1.07k forks source link

DLLs can be missing after update to newer .NET patches #42445

Open johnearnshaw opened 1 year ago

johnearnshaw commented 1 year ago

I run a long-running .NET 6 application in a remote Ubuntu server that generally runs fine for long periods of time (Currently tested to 226 days).

Recently the application became unresponsive and looking into the applications error logging, it was throwing the following Exception while we were requesting data from it:

System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

Restating the application fixed the issue so it's not a major problem, but just thought I'd bring this to your attention as this is the only reference of a similar issue I can find online. The full Exception details I have available can be seen below.

As I say, this is not a major problem but I would prefer my application to run longer without any issue, I thought it quite strange that the application starts and runs long-term without any issues and, after a long while, decides System.Runtime.Serialization.Formatters assembly can't be found. Though a following restart of the application fixes the issue.

It's a strange one!? Any suggestion why this may have occurred would be helpful even if solely for informational purpose...

`2023-10-23-13:36:59.498 Modules.Dashboard.DashboardEventPlaygroundModule Modules.Dashboard.DashboardEventPlaygroundModule threw an exception in callback. getEventOutput System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

File name: 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' at NetTools.Serialization.ReflectionUtils.GetInstanceOf(Type type) at NetTools.Serialization.JsonTools.JsonReader.ParseObject(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.ParseValue(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.FromJson(Type t, String json, StringSerializerFactory serializerFactory, Boolean includePrivates) at NetTools.Serialization.Json.ObjectFromJson(Object this, String jsonString, Boolean includePrivates) at Modules.Dashboard.EventPlayground.Classes.AjaxCallbacks.<>c__DisplayClass2_0.b__0(String[] path, DataRequestEventArgs request, String session, String authorization) at Modules.WebServer.DashboardModule.DashboardModule.ProcessAjaxRequest(String[] path, DataRequestEventArgs request, String session, String authorization), Inner exception:

2023-10-23-13:37:43.263 Modules.Dashboard.DashboardEventPlaygroundModule Modules.Dashboard.DashboardEventPlaygroundModule threw an exception in callback. getEventOutput System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

File name: 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' at NetTools.Serialization.ReflectionUtils.GetInstanceOf(Type type) at NetTools.Serialization.JsonTools.JsonReader.ParseObject(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.ParseValue(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.FromJson(Type t, String json, StringSerializerFactory serializerFactory, Boolean includePrivates) at NetTools.Serialization.Json.ObjectFromJson(Object this, String jsonString, Boolean includePrivates) at Modules.Dashboard.EventPlayground.Classes.AjaxCallbacks.<>c__DisplayClass2_0.b__0(String[] path, DataRequestEventArgs request, String session, String authorization) at Modules.WebServer.DashboardModule.DashboardModule.ProcessAjaxRequest(String[] path, DataRequestEventArgs request, String session, String authorization), Inner exception:`

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-system-runtime See info in area-owners.md if you want to be subscribed.

Issue Details
I run a long-running .NET 6 application in a remote Ubuntu server that generally runs fine for long periods of time (Currently tested to 226 days). Recently the application became unresponsive and looking into the applications error logging, it was throwing the following Exception while we were requesting data from it: `System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.` Restating the application fixed the issue so it's not a major problem, but just thought I'd bring this to your attention as this is the only reference of a similar issue I can find online. The full Exception details I have available can be seen below. As I say, this is not a major problem but I would prefer my application to run longer without any issue, I thought it quite strange that the application starts and runs long-term without any issues and, after a long while, decides `System.Runtime.Serialization.Formatters` assembly can't be found. Though a following restart of the application fixes the issue. It's a strange one!? Any suggestion why this may have occurred would be helpful even if solely for informational purpose... `2023-10-23-13:36:59.498 Modules.Dashboard.DashboardEventPlaygroundModule Modules.Dashboard.DashboardEventPlaygroundModule threw an exception in callback. getEventOutput System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified. File name: 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' at NetTools.Serialization.ReflectionUtils.GetInstanceOf(Type type) at NetTools.Serialization.JsonTools.JsonReader.ParseObject(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack`1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.ParseValue(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack`1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.FromJson(Type t, String json, StringSerializerFactory serializerFactory, Boolean includePrivates) at NetTools.Serialization.Json.ObjectFromJson(Object this, String jsonString, Boolean includePrivates) at Modules.Dashboard.EventPlayground.Classes.AjaxCallbacks.<>c__DisplayClass2_0.b__0(String[] path, DataRequestEventArgs request, String session, String authorization) at Modules.WebServer.DashboardModule.DashboardModule.ProcessAjaxRequest(String[] path, DataRequestEventArgs request, String session, String authorization), Inner exception: 2023-10-23-13:37:43.263 Modules.Dashboard.DashboardEventPlaygroundModule Modules.Dashboard.DashboardEventPlaygroundModule threw an exception in callback. getEventOutput System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified. File name: 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' at NetTools.Serialization.ReflectionUtils.GetInstanceOf(Type type) at NetTools.Serialization.JsonTools.JsonReader.ParseObject(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack`1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.ParseValue(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack`1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.FromJson(Type t, String json, StringSerializerFactory serializerFactory, Boolean includePrivates) at NetTools.Serialization.Json.ObjectFromJson(Object this, String jsonString, Boolean includePrivates) at Modules.Dashboard.EventPlayground.Classes.AjaxCallbacks.<>c__DisplayClass2_0.b__0(String[] path, DataRequestEventArgs request, String session, String authorization) at Modules.WebServer.DashboardModule.DashboardModule.ProcessAjaxRequest(String[] path, DataRequestEventArgs request, String session, String authorization), Inner exception:`
Author: johnearnshaw
Assignees: -
Labels: `area-System.Runtime`, `untriaged`, `needs-area-label`
Milestone: -
danmoseley commented 1 year ago

Is it possible to compare 8.0? (But perhaps it would take months to repro again..) Does the code at the stack trace get exercised successfully in normal operation, or is the failure when it first gets exercised? Is the app deployed self contained or using shared framework? Are there any ApplicationLoadContexts in use? Is it trimmed? Of course, any way to get this to repro on demand would be helpful.

moellerm commented 1 year ago

You could check if an update to dotnet (.NET 6.0 runtime) has been installed recently or if rolling back dotnet, restarting the application and then updating dotnet reproduces the issue. See https://github.com/dotnet/runtime/issues/60144#issuecomment-1782416574 for a similar case. Obviously without further detail, there could be many other reasons.

LukeDearden commented 1 year ago

while installing updates on an Ubuntu 22.04.3 VM I noticed that the dotnet update from https://packages.microsoft.com/ubuntu/22.04/prod repo actually triggered a restart of our dotnet application service

`sudo apt upgrade -y Reading package lists... Done Building dependency tree... Done Reading state information... Done Calculating upgrade... Done The following packages have been kept back: apt apt-utils libapt-pkg6.0 The following packages will be upgraded: aspnetcore-runtime-6.0 dotnet-host dotnet-hostfxr-6.0 dotnet-runtime-6.0 dotnet-runtime-deps-6.0 libprocps8 procps 7 upgraded, 0 newly installed, 0 to remove and 3 not upgraded. 2 standard LTS security updates Need to get 30.0 MB of archives. After this operation, 50.2 kB disk space will be freed. Get:1 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libprocps8 amd64 2:3.3.17-6ubuntu2.1 [36.2 kB] Get:2 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 procps amd64 2:3.3.17-6ubuntu2.1 [378 kB] Get:3 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 dotnet-host amd64 8.0.0-1 [37.7 kB] Get:4 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 dotnet-hostfxr-6.0 amd64 6.0.25-1 [144 kB] Get:5 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 dotnet-runtime-deps-6.0 amd64 6.0.25-1 [2800 B] Get:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 dotnet-runtime-6.0 amd64 6.0.25-1 [22.8 MB] Get:7 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 aspnetcore-runtime-6.0 amd64 6.0.25-1 [6621 kB] Fetched 30.0 MB in 0s (120 MB/s)
(Reading database ... 64881 files and directories currently installed.) Preparing to unpack .../0-libprocps8_2%3a3.3.17-6ubuntu2.1_amd64.deb ... Unpacking libprocps8:amd64 (2:3.3.17-6ubuntu2.1) over (2:3.3.17-6ubuntu2) ... Preparing to unpack .../1-procps_2%3a3.3.17-6ubuntu2.1_amd64.deb ... Unpacking procps (2:3.3.17-6ubuntu2.1) over (2:3.3.17-6ubuntu2) ... Preparing to unpack .../2-dotnet-host_8.0.0-1_amd64.deb ... Unpacking dotnet-host (8.0.0-1) over (7.0.13-1) ... Preparing to unpack .../3-dotnet-hostfxr-6.0_6.0.25-1_amd64.deb ... Unpacking dotnet-hostfxr-6.0 (6.0.25-1) over (6.0.24-1) ... Preparing to unpack .../4-dotnet-runtime-deps-6.0_6.0.25-1_amd64.deb ... Unpacking dotnet-runtime-deps-6.0 (6.0.25-1) over (6.0.24-1) ... Preparing to unpack .../5-dotnet-runtime-6.0_6.0.25-1_amd64.deb ... Unpacking dotnet-runtime-6.0 (6.0.25-1) over (6.0.24-1) ... Preparing to unpack .../6-aspnetcore-runtime-6.0_6.0.25-1_amd64.deb ... Unpacking aspnetcore-runtime-6.0 (6.0.25-1) over (6.0.24-1) ... Setting up dotnet-host (8.0.0-1) ... Setting up dotnet-runtime-deps-6.0 (6.0.25-1) ... Setting up dotnet-hostfxr-6.0 (6.0.25-1) ... Setting up libprocps8:amd64 (2:3.3.17-6ubuntu2.1) ... Setting up dotnet-runtime-6.0 (6.0.25-1) ... Setting up aspnetcore-runtime-6.0 (6.0.25-1) ... Setting up procps (2:3.3.17-6ubuntu2.1) ... Processing triggers for man-db (2.10.2-1) ... Processing triggers for libc-bin (2.35-0ubuntu3.4) ... Scanning processes...
Scanning candidates...
Scanning linux images...

Running kernel seems to be up-to-date.

Restarting services... systemctl restart frends-agent.service`

I checked this out on an older Ubuntu 20.04.6 VM but updating to the same dotnet version from repo https://packages.microsoft.com/ubuntu/20.04/prod did not trigger a service restart

dotnet --list-runtimes Microsoft.AspNetCore.App 6.0.25 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.25 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

arcoba commented 9 months ago

Hi, I've the same issue with my application that start every day at 7.00 AM. It's a console app that running on Ubuntu 22.04 LTS using Microsoft.AspNetCore.App 6.0.27 and Microsoft.NETCore.App 6.0.27 runtimes. The application is framework dependent and deployed as single x64 executable file.

Usually work well, but only sometimes when Newtonsoft.Json.JsonSerializer (13.0.3) try to load System.Runtime.Serialization.Formatters it throw the exception System.IO.FileNotFoundException with message:

"Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified."

I think that this kind of problem was not present until runtime version 6.0.8; is it possible? Do you need any logs or additional information?

Fad.

Sacha-K commented 7 months ago

I saw this one on Windows.

Configuration:

What happened:

In March my method was called about 13k times with website running on version 6.0.27 with no error, failed on the first call after the 6.0.28 update, was called about 8k times with website running on 6.0.28 with no error.

Edit:

I went and read dotnet/runtime#60144 (opened for 2.5 years) and according to comment https://github.com/dotnet/runtime/issues/60144#issuecomment-1001330713 the .NET update forces a restart of websites at a point where it is dotnet 6.0.28 and aspnet-runtime 6.0.27, just after that you get aspnet-runtime 6.0.28 installed and aspnet-runtimes 6.0.27 removed causing the exception.

So either I just accept the single crash after updates, or I "manually" stop website / update / start website instead of letting WU do the updates. :confused:

dotnet-policy-service[bot] commented 4 months ago

Tagging subscribers to this area: @vitek-karas, @agocke, @vsadov See info in area-owners.md if you want to be subscribed.

jeffhandley commented 4 months ago

@agocke This seems more like a Host type issue. What are the expectations for long-running applications where the application is still running when a runtime update is installed?

agocke commented 4 months ago

These symptoms look exactly like what could currently happen during an update (especially the comment about the app crashing at exactly one specific time, when the computer might be scheduled to install updates).

I don't think this is a problem with assembly loading, I think it's a problem with the older framework being uninstalled while the latest patch is being installed. I'm going to move this over to the SDK to further analyze the issue.

FlsZen commented 3 months ago

We've been experiencing issues similar to this seemingly when Windows Update applies a security path. It appears to delete the .NET 6.0.x runtime out from under our running WPF app and relocates the already-loaded assemblies into C:\Config.Msi. I received reports when .NET 6.0.31 was released and again now when 6.0.33 was released.

The issue is specifically that the user performs an operation in the application that they hadn't performed yet (during the process's lifetime) and the runtime needs to load another DLL, but can't find it. For example:

Type: System.IO.FileNotFoundException
Message: Could not load file or assembly 'Accessibility, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.
Source: PresentationFramework
Stack Trace:
   at System.Windows.Controls.Primitives.Popup.PopupSecurityHelper.ForceMsaaToUiaBridge(PopupRoot popupRoot)
   at System.Windows.Controls.Primitives.Popup.CreateWindow(Boolean asyncCall)
   at System.Windows.Controls.Primitives.Popup.OnIsOpenChanged(DependencyObject d, DependencyPropertyChangedEventArgs e)
   at System.Windows.DependencyObject.OnPropertyChanged(DependencyPropertyChangedEventArgs e)
...

and

Could not load file or assembly 'System.IO.Compression.ZipFile, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. The system cannot find the file specified.

I like the concept of running the latest security patch delivered by Windows Update, but is there anything we can do to prevent it from pulling the rug out from under our app? At this point I feel like the best way forward to avoid our software appearing unstable to our customers is to deploy the .NET runtime to a custom folder and run from there.

joeloff commented 2 weeks ago

@FlsZen please take a look at the new option we made available in November. Perhaps these can help address some of the issues you've encountered. By deferring the removal until the next restart, you'll have more control and hopefully reduce crash incidents.

The problems described isn't isolated to the SDK. Unlike .NET Framework that services as part of the OS using CBS that has access to special kernel transactions to deal with in-use files, .NET is serviced through MU. There are a number of factors that impact what happens with running applications and services.

  1. The installer service and restart manager interact with each other. That interaction can be impacted by things like machine policies that will fall back to files-in-use logic that predates Restart Manager / Vista. There are additional nuances, for example, the installer service will do special checks to see whether individual MSIs include the required files-in-use dialog set and if not, modify the actions it takes. This can impact things like the message sent back by the installer service to the installer.
  2. The SIDs associated with the running application/service and the installation. Updates typically run under SYSTEM, but applications do not. Services may run under system, but sometimes they run under dedicated accounts. RM also takes into account the type of application and sessions. It won't stop/restart applications running in a different session and won't touch things when it doesn't have permissions. Critical services are never stopped/restarted.
  3. Applications may or may not have registered with RM for restarts. Even if they did register, if the SIDs don't line up, RM will report restart failures. You'd typically see events similar to "The application SID does not match the conductor SID". Registered applications can provide proper command lines one how to initiate a restart. The default behavior when using something like dotnet.exe to launch an application will not restart properly because RM only detects the host and not the original command line so it ends up executing dotnet.exe on a restart which is effectively a no-op.
  4. Each MSI creates and ends a session with RM. A .NET installation like the runtime consists of multiple MSIs. This means that at the end of each MSI it may try to restart applications only to shut it down immediately when the next MSI executes. The documentation is unclear whether this is asynchronous or whether the installer service waits for RM to complete all the restarts.
  5. Whether or not the .NET update runs with a UI. Messages from the installer services are directed back to the installer engine that may surface additional UI to choose between closing/restarting applications or continue running and restart the machine. When running quietly as is the case in MU, there are defaults that would be selected.