search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.26k stars 4.46k forks source link

Random System.IO.FileNotFoundException for runtime components #97251

Closed prj closed 45 minutes ago

prj commented 4 months ago

Description

All .net projects that run for a certain amount of time (the time varies, it can be from as little as 8 hours to a week) start exhibiting issues with locating runtime components.

Once location of a runtime component fails the issue persists until the application is restarted. After a single failure every single subsequent access to this assembly throw FileNotFoundException. Upon restart everything is fine again.

It happens in many different applications. Both compiled into a single executable as well as exploded ones. It also happens on all the nodes of my cluster.

Reproduction Steps

I am unable to give any reproduction steps, as the issue is sporadic. I would appreciate more information about debugging the issue.

Expected behavior

The application runs normally.

Actual behavior

Examples of various errors:

Jan 12 14:00:43 backend3  XXX[1604551]: fail: Grpc.AspNetCore.Server.ServerCallHandler[6]
Jan 12 14:00:43 backend3  XXX[1604551]:       Error when executing service method 'GetMetadata'.
Jan 12 14:00:43 backend3  XXX[1604551]:       MessagePack.MessagePackSerializationException: Failed to serialize System.Object value.
Jan 12 14:00:43 backend3  XXX[1604551]:        ---> System.TypeInitializationException: The type initializer for 'FormatterCache`1' threw an exception.
Jan 12 14:00:43 backend3  XXX[1604551]:        ---> System.TypeInitializationException: The type initializer for 'FormatterCache`1' threw an exception.
Jan 12 14:00:43 backend3  XXX[1604551]:        ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Reflection.Emit, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
Jan 12 14:00:43 backend3  XXX[1604551]:
Jan 12 14:00:43 backend3  XXX[1604551]:       File name: 'System.Reflection.Emit, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
Jan 12 14:00:43 backend3  XXX[1604551]:          at MessagePack.Resolvers.DynamicUnionResolver.BuildType(Type type)
Jan 12 14:00:43 backend3  XXX[1604551]:          at MessagePack.Resolvers.DynamicUnionResolver.FormatterCache`1..cctor()
Jan 12 14:00:43 backend3  XXX[1604551]:          --- End of inner exception stack trace ---
Jan 12 14:00:43 backend3  XXX[1604551]:          at MessagePack.Resolvers.DynamicUnionResolver.GetFormatter[T]()
Jan 12 14:00:43 backend3  XXX[1604551]:          at MessagePack.Resolvers.StandardResolver.FormatterCache`1..cctor()
Jan 12 14:00:43 backend3  XXX[1604551]:          --- End of inner exception stack trace ---
Jan 12 14:00:43 backend3  XXX[1604551]:          at MessagePack.Resolvers.StandardResolver.GetFormatter[T]()
Jan 12 14:00:43 backend3  XXX[1604551]:          at MessagePack.FormatterResolverExtensions.GetFormatterDynamic(IFormatterResolver resolver, Type type)
Jan 12 14:00:43 backend3  XXX[1604551]:          at MessagePack.Formatters.DynamicObjectTypeFallbackFormatter.Serialize(MessagePackWriter& writer, Object value, MessagePackSerializerOptions options)
Jan 12 14:00:43 backend3  XXX[1604551]:          at MessagePack.MessagePackSerializer.Serialize[T](MessagePackWriter& writer, T value, MessagePackSerializerOptions options)
Jan 12 14:00:43 backend3  XXX[1604551]:          --- End of inner exception stack trace ---
Jan 12 14:00:43 backend3  XXX[1604551]:          at MessagePack.MessagePackSerializer.Serialize[T](MessagePackWriter& writer, T value, MessagePackSerializerOptions options)
Jan 12 14:00:43 backend3  XXX[1604551]:          at MessagePack.MessagePackSerializer.Serialize[T](IBufferWriter`1 writer, T value, MessagePackSerializerOptions options, CancellationToken cancellationToken)
Jan 12 14:00:43 backend3  XXX[1604551]:          at MessagePack.MessagePackSerializer.Serialize[T](Stream stream, T value, MessagePackSerializerOptions options, CancellationToken cancellationToken)
...
Oct 12 06:33:50 backend3 XXX[496324]: fail: Grpc.AspNetCore.Server.ServerCallHandler[6]
Oct 12 06:33:50 backend3 XXX[496324]:       Error when executing service method 'GetMetadata'.
Oct 12 06:33:50 backend3 XXX[496324]:       System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
Oct 12 06:33:50 backend3 XXX[496324]:        ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Security.Cryptography.Encoding, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
Oct 12 06:33:50 backend3 XXX[496324]:
Oct 12 06:33:50 backend3 XXX[496324]:       File name: 'System.Security.Cryptography.Encoding, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertificateValidationCallback remoteCertValidationCallback, SslCertificateTrust trust, ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Security.SslStream.CompleteHandshake(ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
Oct 12 06:33:50 backend3 XXX[496324]:          --- End of inner exception stack trace ---
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
Oct 12 06:33:50 backend3 XXX[496324]:          at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
...
Dec 05 13:37:48 backend3 XXX[948820]:       Connection id "0HMV4HU4USV6L", Request id "0HMV4HU4USV6L:00000002": An unhandled exception was thrown by the application.
Dec 05 13:37:48 backend3 XXX[948820]:       System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Diagnostics.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
Dec 05 13:37:48 backend3 XXX[948820]:
Dec 05 13:37:48 backend3 XXX[948820]:       File name: 'Microsoft.AspNetCore.Diagnostics.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
Dec 05 13:37:48 backend3 XXX[948820]:          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
Dec 05 13:37:48 backend3 XXX[948820]:          at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
Dec 05 13:37:48 backend3 XXX[948820]:          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
Dec 05 13:37:48 backend3 XXX[948820]:          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
Dec 05 13:37:48 backend3 XXX[948820]:          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

There are many more. From HttpClient, to Kestrel to application code.

Regression?

The issue is present in both .NET 5 and .NET 6. I have not tried newer .NET versions yet.

Known Workarounds

No response

Configuration

dotnet --list-runtimes
Microsoft.AspNetCore.App 6.0.26 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.26 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]

ARM64 architecture

uname -a
Linux backend3 5.15.0-1040-oracle #46-Ubuntu SMP Fri Jul 14 21:47:21 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

The deployment is in OCI Cloud.

Other information

Because the applications are run on OCI on virtual machines, and the virtual machines have network attached storage, it could be possible that some requests to the storage fail sporadically.

Does .NET cache the file accesses? So if it fails once then it fails forever? Could this be the issue?

jkotas commented 4 months ago

Does .NET cache the file accesses?

Yes, it does. The most likely explanation of the behavior that you are seeing is that this cache is corrupted. The cache is implemented in C/C++ runtime code.

Do you see other a crashes like segmentation faults too? They may have same root cause and they may be easier to debug.

Do your projects use any 3rd party packages with native code or telemetry solutions? They may be the cause of the problem.

ghost commented 4 months ago

Tagging subscribers to this area: @vitek-karas, @agocke, @vsadov See info in area-owners.md if you want to be subscribed.

Issue Details
### Description All .net projects that run for a certain amount of time (the time varies, it can be from as little as 8 hours to a week) start exhibiting issues with locating runtime components. Once location of a runtime component fails the issue persists until the application is restarted. After a single failure every single subsequent access to this assembly throw FileNotFoundException. Upon restart everything is fine again. It happens in many different applications. Both compiled into a single executable as well as exploded ones. It also happens on all the nodes of my cluster. ### Reproduction Steps I am unable to give any reproduction steps, as the issue is sporadic. I would appreciate more information about debugging the issue. ### Expected behavior The application runs normally. ### Actual behavior Examples of various errors: ``` Jan 12 14:00:43 backend3 XXX[1604551]: fail: Grpc.AspNetCore.Server.ServerCallHandler[6] Jan 12 14:00:43 backend3 XXX[1604551]: Error when executing service method 'GetMetadata'. Jan 12 14:00:43 backend3 XXX[1604551]: MessagePack.MessagePackSerializationException: Failed to serialize System.Object value. Jan 12 14:00:43 backend3 XXX[1604551]: ---> System.TypeInitializationException: The type initializer for 'FormatterCache`1' threw an exception. Jan 12 14:00:43 backend3 XXX[1604551]: ---> System.TypeInitializationException: The type initializer for 'FormatterCache`1' threw an exception. Jan 12 14:00:43 backend3 XXX[1604551]: ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Reflection.Emit, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified. Jan 12 14:00:43 backend3 XXX[1604551]: Jan 12 14:00:43 backend3 XXX[1604551]: File name: 'System.Reflection.Emit, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' Jan 12 14:00:43 backend3 XXX[1604551]: at MessagePack.Resolvers.DynamicUnionResolver.BuildType(Type type) Jan 12 14:00:43 backend3 XXX[1604551]: at MessagePack.Resolvers.DynamicUnionResolver.FormatterCache`1..cctor() Jan 12 14:00:43 backend3 XXX[1604551]: --- End of inner exception stack trace --- Jan 12 14:00:43 backend3 XXX[1604551]: at MessagePack.Resolvers.DynamicUnionResolver.GetFormatter[T]() Jan 12 14:00:43 backend3 XXX[1604551]: at MessagePack.Resolvers.StandardResolver.FormatterCache`1..cctor() Jan 12 14:00:43 backend3 XXX[1604551]: --- End of inner exception stack trace --- Jan 12 14:00:43 backend3 XXX[1604551]: at MessagePack.Resolvers.StandardResolver.GetFormatter[T]() Jan 12 14:00:43 backend3 XXX[1604551]: at MessagePack.FormatterResolverExtensions.GetFormatterDynamic(IFormatterResolver resolver, Type type) Jan 12 14:00:43 backend3 XXX[1604551]: at MessagePack.Formatters.DynamicObjectTypeFallbackFormatter.Serialize(MessagePackWriter& writer, Object value, MessagePackSerializerOptions options) Jan 12 14:00:43 backend3 XXX[1604551]: at MessagePack.MessagePackSerializer.Serialize[T](MessagePackWriter& writer, T value, MessagePackSerializerOptions options) Jan 12 14:00:43 backend3 XXX[1604551]: --- End of inner exception stack trace --- Jan 12 14:00:43 backend3 XXX[1604551]: at MessagePack.MessagePackSerializer.Serialize[T](MessagePackWriter& writer, T value, MessagePackSerializerOptions options) Jan 12 14:00:43 backend3 XXX[1604551]: at MessagePack.MessagePackSerializer.Serialize[T](IBufferWriter`1 writer, T value, MessagePackSerializerOptions options, CancellationToken cancellationToken) Jan 12 14:00:43 backend3 XXX[1604551]: at MessagePack.MessagePackSerializer.Serialize[T](Stream stream, T value, MessagePackSerializerOptions options, CancellationToken cancellationToken) ... ``` ``` Oct 12 06:33:50 backend3 XXX[496324]: fail: Grpc.AspNetCore.Server.ServerCallHandler[6] Oct 12 06:33:50 backend3 XXX[496324]: Error when executing service method 'GetMetadata'. Oct 12 06:33:50 backend3 XXX[496324]: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. Oct 12 06:33:50 backend3 XXX[496324]: ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Security.Cryptography.Encoding, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified. Oct 12 06:33:50 backend3 XXX[496324]: Oct 12 06:33:50 backend3 XXX[496324]: File name: 'System.Security.Cryptography.Encoding, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertificateValidationCallback remoteCertValidationCallback, SslCertificateTrust trust, ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Security.SslStream.CompleteHandshake(ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) Oct 12 06:33:50 backend3 XXX[496324]: --- End of inner exception stack trace --- Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) Oct 12 06:33:50 backend3 XXX[496324]: at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) Oct 12 06:33:50 backend3 XXX[496324]: at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) ... ``` ``` Dec 05 13:37:48 backend3 XXX[948820]: Connection id "0HMV4HU4USV6L", Request id "0HMV4HU4USV6L:00000002": An unhandled exception was thrown by the application. Dec 05 13:37:48 backend3 XXX[948820]: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Diagnostics.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified. Dec 05 13:37:48 backend3 XXX[948820]: Dec 05 13:37:48 backend3 XXX[948820]: File name: 'Microsoft.AspNetCore.Diagnostics.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60' Dec 05 13:37:48 backend3 XXX[948820]: at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi) Dec 05 13:37:48 backend3 XXX[948820]: at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) Dec 05 13:37:48 backend3 XXX[948820]: at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi) Dec 05 13:37:48 backend3 XXX[948820]: at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task) Dec 05 13:37:48 backend3 XXX[948820]: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) ``` There are many more. From HttpClient, to Kestrel to application code. ### Regression? The issue is present in both .NET 5 and .NET 6. I have not tried newer .NET versions yet. ### Known Workarounds _No response_ ### Configuration ``` dotnet --list-runtimes Microsoft.AspNetCore.App 6.0.26 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.26 [/usr/lib/dotnet/shared/Microsoft.NETCore.App] ``` ARM64 architecture ``` uname -a Linux backend3 5.15.0-1040-oracle #46-Ubuntu SMP Fri Jul 14 21:47:21 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux ``` The deployment is in OCI Cloud. ### Other information Because the applications are run on OCI on virtual machines, and the virtual machines have network attached storage, it could be possible that some requests to the storage fail sporadically. Does .NET cache the file accesses? So if it fails once then it fails forever? Could this be the issue?
Author: prj
Assignees: -
Labels: `area-AssemblyLoader-coreclr`, `untriaged`, `needs-area-label`
Milestone: -
prj commented 3 months ago

No, I do not see any other issues, no segmentation faults or anything of that nature. The only native library I use is the GRPC runtime, and this issue only happens on the OCI nodes. It did not happen on a dedicated linux server running the same applications.

So it is something that is triggered by the environment. All of my apps are doing it as well, and they are all vastly different.

Do the assemblies get garbage collected and then re-loaded at a later time? The main issue is that once loading of an assembly fails it stays in this failed state until restarted, so it will throw FileNotFound exceptions for every subsequent request (or attempt to access to that assembly). Is there any configuration I can try to prevent this behaviour?

My problem is that it is a production cluster, so I am in a bit of a pickle.

jkotas commented 3 months ago

Do the assemblies get garbage collected and then re-loaded at a later time?

By default, the assemblies are loaded once and stay loaded until the process exits.

It is possible to load assemblies as collectible using https://learn.microsoft.com/en-us/dotnet/api/system.runtime.loader.assemblyloadcontext.-ctor. Do your apps use custom AssemblyLoadContexts?

prj commented 3 months ago

No, they do not. I feel like I could reproduce this issue even with just a blank aspnetcore sample app.

I mean the errors get thrown even from the Kestrel runtime, which does not even have anything to do with any apps.

P.S. Maybe it is worth saying all these apps are aspnetcore with kestrel serving the requests, if that changes anything. But I am really not doing anything fancy. Some of the apps have no frontend at all, they are purely grpc service providers that hit a mysql database. It does not seem that what the app does has any influence on this at all, since all the apps get these errors...

prj commented 3 months ago

This happened again today. 3 separate nodes all on the same day.

It might be some kind of issues on OCI with their storage. The big problem is that the FileNotFound errors keep repeating over and over once the issue happens a single time.

Because on OCI you do not have instance local storage, I am thinking to create a ramdisk and place the runtime on the ramdisk. This should hopefully alleviate the issue.

agocke commented 4 weeks ago

@prj Any update? Based on your previous symptoms it sounds likely a storage issue.

dotnet-policy-service[bot] commented 4 weeks ago

This issue has been marked needs-author-action and may be missing some important information.

dotnet-policy-service[bot] commented 2 weeks ago

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

dotnet-policy-service[bot] commented 45 minutes ago

This issue will now be closed since it had been marked no-recent-activity but received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.