Running a simple ASP.NET core application causing Coreclr.dll throwing exception

[mpiracha]

I am testing .NET 6 on my dev environment. The simple hello world console app works well, but simple ASP.NET Core Web application consistently throwing following exception when running on Windows Server 2016. I had the same app running well on different EC2 with similar configuration. I need some help to understand what's causing the issue before I prepare/deploy application on production.

Thanks.

From Windows Event Logs:

Faulting application name: WebTestApp6.exe, version: 1.0.0.0, time stamp: 0x62571213 Faulting module name: coreclr.dll, version: 6.0.522.21309, time stamp: 0x625708f4 Exception code: 0xc0000005 Fault offset: 0x0000000000082288 Faulting process id: 0x44c Faulting application start time: 0x01d87055e87be788 Faulting application path: D:\TestWeb2\WebTestApp6.exe Faulting module path: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.5\coreclr.dll Report Id: a368f0c5-d5c3-446f-b79a-b92b1fa6df3f Faulting package full name: Faulting package-relative application ID:

Stack trace:

D:\TestWeb2>WebTestApp6.exe
Fatal error. Internal CLR error. (0x80131506)
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Span`1<System.Object> ByRef, System.Signature, Boolean, Boolean)
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at System.Reflection.CustomAttribute.AddCustomAttributes(ListBuilder`1<System.Object> ByRef, System.Reflection.RuntimeModule, Int32, System.RuntimeType, Boolean, ListBuilder`1<System.Object>)
   at System.Reflection.CustomAttribute.GetCustomAttributes(System.Reflection.RuntimeModule, Int32, Int32, System.RuntimeType)
   at System.Reflection.CustomAttribute.GetCustomAttributes(System.RuntimeType, System.RuntimeType, Boolean)
   at System.Attribute.GetCustomAttributes(System.Reflection.MemberInfo, System.Type, Boolean)
   at System.Attribute.GetCustomAttribute(System.Reflection.MemberInfo, System.Type, Boolean)
   at System.Diagnostics.Tracing.EventSource.GetCustomAttributeHelper(System.Reflection.MemberInfo, System.Type, System.Diagnostics.Tracing.EventManifestOptions)
   at System.Diagnostics.Tracing.EventSource.GetGuid(System.Type)
   at System.Diagnostics.Tracing.EventSource..ctor(System.Diagnostics.Tracing.EventSourceSettings, System.String[])
   at System.Net.NetEventSource..cctor()
   at System.Net.Quic.Implementations.MsQuic.Internal.MsQuicApi..cctor()
   at System.Net.Quic.Implementations.MsQuic.MsQuicImplementationProvider.get_IsSupported()
   at Microsoft.AspNetCore.Hosting.WebHostBuilderQuicExtensions.UseQuic(Microsoft.AspNetCore.Hosting.IWebHostBuilder)
   at Microsoft.AspNetCore.Hosting.WebHostBuilderKestrelExtensions.UseKestrel(Microsoft.AspNetCore.Hosting.IWebHostBuilder)
   at Microsoft.AspNetCore.WebHost.ConfigureWebDefaults(Microsoft.AspNetCore.Hosting.IWebHostBuilder)
   at Microsoft.Extensions.Hosting.GenericHostBuilderExtensions+<>c__DisplayClass0_0.<ConfigureWebHostDefaults>b__0(Microsoft.AspNetCore.Hosting.IWebHostBuilder)
   at Microsoft.Extensions.Hosting.GenericHostWebHostBuilderExtensions.ConfigureWebHost(Microsoft.Extensions.Hosting.IHostBuilder, System.Action`1<Microsoft.AspNetCore.Hosting.IWebHostBuilder>, System.Action`1<Microsoft.Extensions.Hosting.WebHostBuilderOptions>)
   at Microsoft.AspNetCore.Builder.WebApplicationBuilder..ctor(Microsoft.AspNetCore.Builder.WebApplicationOptions, System.Action`1<Microsoft.Extensions.Hosting.IHostBuilder>)
   at Microsoft.AspNetCore.Builder.WebApplication.CreateBuilder(System.String[])
   at Program.<Main>$(System.String[])

EC2 instance and OS version:

OS Name: Microsoft Windows Server 2016 Datacenter OS Version: 10.0.14393 N/A Build 14393 OS Manufacturer: Microsoft Corporation OS Configuration: Member Server OS Build Type: Multiprocessor Free Registered Owner: EC2 Registered Organization: Amazon.com Product ID: 00376-40000-00000-AA849 Original Install Date: 4/12/2019, 10:46:23 AM System Boot Time: 5/24/2022, 4:38:08 PM System Manufacturer: Xen System Model: HVM domU System Type: x64-based PC Processor(s): 1 Processor(s) Installed. [01]: Intel64 Family 6 Model 79 Stepping 1 GenuineIntel ~2300 Mhz BIOS Version: Xen 4.2.amazon, 8/24/2006

[dotnet-issue-labeler[bot]]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[Beau-Gosse-dev]

@mpiracha since this looks like a sample ASP.NET Core Web application, can you share the code here so we can try to reproduce? Also you said "I had the same app running well on different EC2 with similar configuration." what if anything was different about that EC2 instance?

[mpiracha]

WebTestApp6.zip @Beau-Gosse-dev , the sample attached.

The samples are just out-of-the-box project files. I have one console app that runs without any issue. The ASP.NET 6 is crashing though. Self contain option also crashed if it helps.

All EC2 instances have same OS and May CU applied. I have also stopped AV just in case. Otherwise, there are unfortunately no obvious differences that I can find.

[Beau-Gosse-dev]

@mpiracha I just tried to reproduce this with your code on a new Windows Server 2016 EC2 instance with the latest version of .NET and it worked just fine. I'm not sure what to suggest other than updating everything and maybe starting over from scratch with another dotnet new webapp from the command line.

[birojnayak]

@Beau-Gosse-dev @mpiracha looks like this Microsoft.AspNetCore.Hosting.WebHostBuilderQuicExtensions.UseQuic is experimental... I don't see that in the attached code...

[mpiracha]

@Beau-Gosse-dev, thank you for helping me out. The dotnet new webapp from CLI did not change any thing :(.

Let me go a look for any other differences between my EC2 instances.

[mpiracha]

@birojnayak, The related assemblies are in C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.5 folder. Do you feel they need to be in some other location?

[mpiracha]

FYI

The QuicImplementationProviders.Default.IsSupported call from System.Net.Quic causing the issue.

See the attached code.

msquictest.zip

[Beau-Gosse-dev]

@mpiracha Are you saying that this sample console app throws the same error? Does it only throw on that single EC2 instance? Running on Windows 10 it doesn't give any errors for me.

[mpiracha]

@Beau-Gosse-dev, yes, so far its crashing only on couple of Windows Server 2016. My local Win10 machine has no issues.

Also, I found some discussion around Quic issue here: https://github.com/dotnet/aspnetcore/pull/40376 in case you find it helpful.

[mpiracha]

@Beau-Gosse-dev, @birojnayak

Some more details. We were able to narrow down the issue to specific area - the attribute [EventSource(Name = "Derived.EventSource")] is the culprit.

The attached sample will run fine, if compiled under .NET5 but produce the same internal error when compiled for NET6.

NetEventSourceTest.zip

I hope this provides some useful insight to the issue.

[mpiracha]

@jeffschwMSFT

One additional piece of information.

Out of two servers, where I have this exception, one server, runs fine under win-x86.

[mangod9]

@mpiracha are you able to capture a dump with the failure? That would help with diagnosing further. Thx!

[ChernayaKoshka]

@mpiracha if you are using certain security agents (e.g. Contrast .NET) they may be interfering. I was having an extremely similar issue to you and it turns out our Contrast .NET agent was the version prior to adding .NET 6 support. Upgrading the agent resolved this mysterious crash.

If you aren't, I hope whoever else comes across this post will find it useful.

[danmoseley]

@rameshreddy5763 hit this in dotnet/aspnetcore#46348.

@rameshreddy5763 do you have any extra software on the OS such as security agents, 3rd party virus checkers/anti malware, etc? If so can you try again after disabling those (or just on a fresh OS image) since it appears to have helped others above.

[rameshreddy5763]

We have Symantec running on the machine. Let me disable and try again

[danmoseley]

one more thing @rameshreddy5763 -- can you try running just the application linked by @mpiracha here? https://github.com/dotnet/runtime/issues/69805#issuecomment-1139093838

essentially just create a console app and paste this code in

using System;
using System.Diagnostics.Tracing;

namespace NetEventSourceTest
{
    [EventSource(Name = "Derived.EventSource")]
    internal sealed partial class DerivedEventSource : EventSource
    {
    }

    class Program
    {
        static void Main(string[] args)
        {
            Guid guid = EventSource.GetGuid(typeof(DerivedEventSource));
            Console.WriteLine($"{nameof(DerivedEventSource)} {guid}");
        }
    }
}

That works fine for me, but apparently on machines where this issue appears, it does not.

[rameshreddy5763]

@danmoseley

This code works fine in local machine, but its failing on machines where this issue appears.

[zhangbest5]

I found problem,You need to uninstall all security software or test on a new server. The problem will be solved.

[zhangbest5]

security software is the Key!