InvalidProgramException in 1.9.0 in .NET Core 2.2 / Fargate

[lpodolak]

Today I spot again "System.InvalidProgramException: Common Language Runtime detected an invalid program" errors in our logs

To Reproduce I don't know how to reproduce it in a deterministic way. It is the first time I spot the error since upgrading to 1.9.0 of Datadog.Trace.ClrProfiler.Managed

Expected behavior This exception should not happen.

Screenshots

System.InvalidProgramException: Common Language Runtime detected an invalid program.
   at Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.ResultNext<TFilter, TFilterAsync>(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
   at async Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultFilters()
   at async Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at void Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at void Rethrow(object)
   at void Datadog.Trace.ClrProfiler.Integrations.AspNetCoreMvc2Integration.Rethrow(object context, int opCode, int mdToken, long moduleVersionPtr)
   at Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
   at async Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at async Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at async Task Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)
   at async Task Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)
   at async Task Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
   at async Task <internal-middleware3>:line 39
   at async Task <internal-middleware2>:line 29
   at async Task <internal-middleware1>:line 33
   at async Task Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)

Runtime environment:

• Instrumentation mode: manual with NuGet package
• Tracer version: 1.9.0
• OS: mcr.microsoft.com/dotnet/core/aspnet:2.2.3-stretch-slim (docker image) @ Fargate v1.3 (LATEST)
• CLR: .NET Core 2.2.3

Additional context We had few serious problems with InvalidProgramException while running with 1.7.0. We upgraded to 1.9.0 on 14th November and today (Nov 20, 2019 @ 13:37:54.901 UTC) I saw this error for the first time since upgrade. Since first failure, it made that particular Fargate task dead, responding with such error to all incoming requests.

Please let me know if you need any more details from me.

[lucaspimentel]

Thanks @lpodolak for reporting this.

You mentioned that you updated the Datadog.Trace.ClrProfiler.Managed NuGet package to 1.9.0. The NuGet package is no longer required in .NET Core applications since release 1.8.0. Did you also update the profiler package? That's the one distributed in rpm/deb/tag.gz files. Once you update the profiler package to 1.9.0, you don't need the NuGet package anymore and you can remove it from your application. If you are using manual instrumentation to enhance the automatic instrumentation, you can use the base Datadog.Trace package.

I also noted you are running on .NET Core 2.2.3. The JIT compiler had a few bugs that affected our instrumentation. One in particular was fixed in 2.2.6. Can you test instrumenting your app in a newer release of .NET Core? .NET Core 2.2.8 was just released yesterday so you should be able to use mcr.microsoft.com/dotnet/core/aspnet:2.2.8-stretch-slim.

[lpodolak]

Thank you for your comments @lucaspimentel - I’ll try to apply them next week. Is it also safe to upgrade to net core 3.0?

[lucaspimentel]

Automatic instrumentation for most supported libraries (SqlClient, StackExchange.Redis, etc) works on .NET Core 3.0, but support for instrumenting ASP.NET Core MVC 3 (the web framework) will be added in the next release.

[BryceBarbara]

@lucaspimentel

Do we have any sort of timeline on when that might be released? I'm just trying to weigh going down the route of manually adding tracing or waiting for the release.

[colin-higgins]

We are looking to release tomorrow (Nov 22nd).

[lpodolak]

thanks @lucaspimentel - I was away last week. Today I will upgrade our runtime to 2.2.8. You can close the issue - I will report the problem if I ever see the the bug again. I appreciate your help a lot!

[lpodolak]

Unfortunately, I have to report again that we had InvalidProgramExceptions still few times after upgrade to .net core 2.2.8 runtime, so this is not a workaround. Right now, instead of finding a fix for this (from dd or clr team?) I try to make sure we can recover from such failures.