search

OrchardCMS / OrchardCore

Orchard Core is an open-source modular and multi-tenant application framework built with ASP.NET Core, and a content management system (CMS) built on top of that framework.
https://orchardcore.net
BSD 3-Clause "New" or "Revised" License
7.23k stars 2.34k forks source link

Prevent `ShellDescriptorManager` exception from bringing down the app #7232

Open themattkenney opened 3 years ago

themattkenney commented 3 years ago

On one of our Tenants, that is running SQL Lite we had a System.NullReferenceException causing the whole App Pool to lockup requiring an app pool restart.

We are looking into what caused the issue however I would expect an error on one tenant to not lockup the app pool.

Issue Details from app insights: image

Call Stack:

System.NullReferenceException:
   at YesSql.QueryExtensions.Query (YesSql.Abstractions, Version=0.0.0.0, Culture=neutral, PublicKeyToken=51cdb8c1938e4789)
   at OrchardCore.Environment.Shell.Data.Descriptors.ShellDescriptorManager+<GetShellDescriptorAsync>d__8.MoveNext (OrchardCore.Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullOrchardCore.Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\orchardcore\src\OrchardCore\OrchardCore.Infrastructure\Shell\ShellDescriptorManager.csOrchardCore.Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Environment.Shell.Builders.ShellContextFactory+<OrchardCore-Environment-Shell-Builders-IShellContextFactory-CreateShellContextAsync>d__5.MoveNext (OrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\orchardcore\src\OrchardCore\OrchardCore\Shell\Builders\ShellContextFactory.csOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 42)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Environment.Shell.ShellHost+<GetOrCreateShellContextAsync>d__12.MoveNext (OrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\orchardcore\src\OrchardCore\OrchardCore\Shell\ShellHost.csOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 87)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Environment.Shell.ShellHost+<GetScopeAsync>d__13.MoveNext (OrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\orchardcore\src\OrchardCore\OrchardCore\Shell\ShellHost.csOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 118)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at OrchardCore.Modules.ModularTenantContainerMiddleware+<Invoke>d__4.MoveNext (OrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\orchardcore\src\OrchardCore\OrchardCore\Modules\ModularTenantContainerMiddleware.csOrchardCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 49)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware+<Invoke>d__9.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=3.1.8.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
hishamco commented 3 years ago

/cc @jtkech

sebastienros commented 3 years ago

It might be an issue in ShellDescriptorManager. But ideally YesSql should have thrown an ArgumentException if the inputs were not valid. That would help.

themattkenney commented 3 years ago

Update: We were able to figure out why we got the System.NullReferenceException. The issue was in the appsetting.json file, we removed all the unused tenants from our site but missed one tenant. This caused the 500 error when a bot called the site, we removed this tenant, and now no System.NullReferenceException.

@sebastienros if we wanted to block the error from crashing the app pool where would we start?

Skrypt commented 3 years ago

I think you want to wait on @jtkech answer on that one 😄

sebastienros commented 3 years ago

@themattkenney would be awesome to have just a little more details, to be able to repro. Like what to remove if we have two running tenants to get to this issue.

Skrypt commented 3 years ago

I'd guess removing the tenant folder or simply the SQLite database while keeping the tenant config could potentially cause this issue? If not, then maybe the tenant that contains all the other tenants!

jtkech commented 3 years ago

@themattkenney

Maybe in GetShellDescriptorAsync(), add a try catch block around

_shellDescriptor = await _session.Query<ShellDescriptor>().FirstOrDefaultAsync();

So that it returns a null _shellDescriptor and then fallback to a MinimumShellDescriptor, but without failing.

Let us know if it fixes your issue

themattkenney commented 3 years ago

We will give this a try! Thank you

Piedone commented 1 month ago

Can you share the result @themattkenney?