Open Shazwazza opened 7 years ago
No idea what causes it, but seeing it on other Cloud sites. Looking into it.
Another log:
2017-03-07 08:03:50,783 [P23288/D10/T195] ERROR Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Failed to build models.
System.Threading.ThreadAbortException: Thread was being aborted.
at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken)
at System.Web.Compilation.BuildManager.GetBuildResultFromCacheInternal(String cacheKey, Boolean keyFromVPP, VirtualPath virtualPath, Int64 hashCode, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVPathBuildResultFromCacheInternal(VirtualPath virtualPath, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVPathBuildResult(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetCompiledAssembly(String virtualPath)
at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.GetModelsAssembly(Boolean forceRebuild)
at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.EnsureModels()
Because the exception comes from BuildManager.GetCompiledAssembly
it means that we already went through the csharp code generation and are now compiling the code, and for some reason BuildManager
fails due to... timeout?
Looking at BuildManager
source code this happens when:
lock(this) {
...
So, locking the BuildManager
instance.
Readings:
These explain what happens when a thread aborts, but not why a thread would abort while trying to ReliableEnter
a monitor, though. I don't think it's due to ReliableEnter
though - it's probably waiting for the lock and is being aborted by something else (ASP.NET timeout?).
Notes:
It could be that the thread is being aborted due to timeout, depending on the value of
<httpRuntime executionTimeout="????" />
in web.config
. MSDN seems to imply that the default value is 110s but what's the value on Cloud?
About the ThreadAbortException
: pretty sure it's due to a timeout, see U4-9595 which should detect when it's a timeout and log a more explicit message (due in 7.5.12).
Now need to figure out why we end up in a timeout, especially here in BuildManager
. Wondering if that might be linked to some deadlock in our code. Investigating (see U4-9606).
See PR https://github.com/zpqrtbnk/Zbu.ModelsBuilder/pull/136 which tries to refactor the locks in PureLiveModelFactory
to reduce the possibility of deadlocks. But TBH I cannot figure out exactly what's causing the timeout, and it's super hard to troubleshoot without repro.
@zpqrtbnk We are receiving this in Umbraco 7.5.4. We are not sure what triggers this, as no deployments or doc type updates are being made prior to these log entries, however we do have several scheduled tasks (content syncs) that are called frequently via custom surface controller URLs that could possibly be interfering somehow if they happen to run at the same time as the model builder? In our specific Azure web app environment, this (deadlock?) is actually causing the entire site (including the umbraco backend) to become unresponsive until we restart the site. Note that these messages occur every minute until the reset.
2017-03-21 12:24:25,918 [P6588/D6/T14] ERROR Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Failed to build models.
System.Threading.ThreadAbortException: Thread was being aborted.
at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken)
at System.Web.Compilation.BuildManager.GetBuildResultFromCacheInternal(String cacheKey, Boolean keyFromVPP, VirtualPath virtualPath, Int64 hashCode, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVPathBuildResultFromCacheInternal(VirtualPath virtualPath, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetVPathBuildResult(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean ensureIsUpToDate)
at System.Web.Compilation.BuildManager.GetCompiledAssembly(String virtualPath)
at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.GetModelsAssembly(Boolean forceRebuild)
at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.EnsureModels()
2017-03-21 12:24:26,012 [P6588/D6/T14] WARN Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory - Running without models.
I have merged the PR, which should be more robust WRT locks. Also, next version of Core (7.5.12) will ship with a feature that auto-creates a (limited) number of process minidumps on ThreadAbortException
due to a timeout. The idea is to be able to analyse these minidumps and understand what is causing the deadlocks.
Minidumps should show in ~/App_Data/MiniDump
.
I'm keeping this issue open.
Not sure how to replicate but seen it on my site a few times when deploying and something in the /bin changes.
Here's the log which shows the initial /bin change, then it will YSOD, i changed the web.config debug to 'true', then it got 'stuck' for about 5 mins, assuming it's waiting for MainDom, then eventually it boots.