WebAPI project reporting ActivationException randomly after registrations work for long periods

AdamBowler commented 3 years ago

Describe the bug

We're experiencing an issue in a WebAPI project where our DI will work for long periods of time on live but at random intervals, and it could be weeks between occurrences, we start getting activation exceptions. If we recycle the affected app pool then it goes away and the DI continues as normal.

Expected behavior

The DI should work constantly.

Actual behavior

The DI works for a period of time and then the following errors are thrown repeatedly until we recycle the affected app pool,

System.AggregateException: One or more errors occurred. ---> SimpleInjector.ActivationException: No registration for type ISomething could be found.

   at SimpleInjector.Container.ThrowMissingInstanceProducerException(Type type)
   at SimpleInjector.Container.GetInstanceFromProducer(InstanceProducer instanceProducer, Type serviceType)
   at SimpleInjector.Container.GetInstance[TService]()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at Library.SomeClass.<>c__DisplayClass127_0.<GetSomething>b__0() in D:\SomeFoldersHere\SomeClass.cs:line 2033
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Library.SomeRepo.DoSomething()
   at Library.SomeRepo.DoSomethingElse()
   at Library.Controllers.SomeController.GetSomething()
   at lambda_method(Closure , Object , Object[] )
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass10.<GetExecutor>b__9(Object instance, Object[] methodParameters)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()
---> (Inner Exception #0) SimpleInjector.ActivationException: No registration for type IReceiptStorage could be found.
   at SimpleInjector.Container.ThrowMissingInstanceProducerException(Type type)
   at SimpleInjector.Container.GetInstanceFromProducer(InstanceProducer instanceProducer, Type serviceType)
   at SimpleInjector.Container.GetInstance[TService]()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at Library.SomeClass.<>c__DisplayClass127_0.<GetSomething>b__0() in D:\SomeFoldersHere\SomeClass.cs:line 2033
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()

To Reproduce

We have been unable to find what causes the issue to occur but it only occurs in our WebAPI project and nowhere else across our stack and product range.

Additional context

Add any other context about the problem here. e.g.:

SimpleInjector v5.0.3
SimpleInjector.Integration.WebApi v5.0.0
.NET Framework v4.6.1

dotnetjunkie commented 3 years ago

Did you configure dynamic assembly compilation in Simple Injector using container.Options.EnableDynamicAssemblyCompilation()? If so, you might want to turn it off and try again.

dotnetjunkie commented 3 years ago

Are you using Auto-Registration by calling AppDomain.CurrentDomain.GetAssemblies() in your application startup? That might be the problem. Try using BuildManager.GetReferencedAssemblies().Cast<Assembly>() instead, because assemblies are only loaded on demand when IIS recycles. Also see this.

AdamBowler commented 3 years ago

We aren't using container.Options.EnableDynamicAssemblyCompilation() and neither are we using AppDomain.CurrentDomain.GetAssemblies()

AdamBowler commented 3 years ago

We found no correlation between IIS recycles causing the ActivationException in our investigation, we also checked to see if the error occurred after a period of no traffic but it was occurring randomly during high traffic periods as well, so I believe that title change may be misleading

dotnetjunkie commented 3 years ago

Can you show the code that registers IReceiptStorage?

AdamBowler commented 3 years ago

container.Register<IReceiptStorageTypeFactory, ReceiptStorageTypeFactory>(Lifestyle.Singleton);
container.Register<IReceiptStorageFactory, SelenityReceiptStorageFactory>(Lifestyle.Singleton);
container.Register(
                () => container.GetInstance<IReceiptStorageFactory>()
                               .Create(),
                Lifestyle.Singleton
            );

These bottom registration is for IReceiptStorage

dotnetjunkie commented 3 years ago

The exception message presented in your initial post, is that the complete exception message, or did you cut some details from that message? If so, can you post those details?

AdamBowler commented 3 years ago

That's the full exception message just with some folder structure removed and a couple of classes renamed

dotnetjunkie commented 3 years ago

The registrations you showed are not dynamic, and considering the given exception message, this means that the Container instance that IReceiptStorage is resolved from is:

Not empty (an empty Container instance throws a message containing the text "Please note that the container instance you are resolving from contains no registrations.")
But does not contain the registration for IReceiptStorage.

Considering the fact that a Simple Injector Container gets locked once in use and can't be changed nor cleared, my assumption is that somewhere in your code base, an accidental new Container instance is created that doesn't get initialized with IReceiptStorage.

You can try the following in order to get more information when the exception happens:

When you call Register on a locked container, the container throws an exception containing the stack trace of the location where that instance was locked. You might be able to use that information to diagnose the problem.
If there are multiple Container instances, each instance will have its own hash code, which you can retrieve using GetHashCode(). Might be useful to log the Container's hashcode at startup and log the hash code of the Container instance throwing the exception.
You can call Container.GetRegistrations to get a list of all registrations that are in the container. You can log that information upon failure in order to get sense of what's the difference between the 'real' container and the 'phantom' container instance.

But there might be another reason for this failure. Although Simple Injector is designed for concurrent use, it is still possible that you stumbled upon an unfortunate concurrency bug within Simple Injector code base, but this is impossible for me to analyze without more information from your part. If the currency bug is the cause, you will see the problem to likely go away by calling container.Verify() upon startup. Calling Verify() pre-compiles all object graphs in a single thread. This might cause the application to startup more slowly, but if there is an bug caused by some parallel operation, the call to Verify() will likely solve this. Downside of you calling Verify() to solve the problem is that in that case we won't find the cause of the concurrency bug, and we won't be able to solve it.

dotnetjunkie commented 3 years ago

Hi @AdamBowler, any luck in finding the source of the problem?

simpleinjector / SimpleInjector