NuCache panic issue on deploy

[bjarnef]

When deploying content from e.g. local to live enviroment, it seems the deploy fails and the live environment shows an ysod.

Tested on Umbraco v8.1.1 and Deploy v3.1.1 and using ModelsBuilder dll mode.

On local it seems the deploy stop because the remote environment throw an ysod.

EXCEPTION:

Umbraco.Deploy.Exceptions.RemoteApiException: The remote API has thrown an exception.
   at Umbraco.Deploy.Environments.RemoteUmbracoEnvironment.<SucceedOrThrow>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Umbraco.Deploy.Environments.RemoteUmbracoEnvironment.<WaitAction>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Umbraco.Deploy.Environments.RemoteUmbracoEnvironment.<CompleteSessionAsync>d__29.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Umbraco.Deploy.Work.WorkItems.DeployRestoreWorkItemBase.<CloseSessionOn>d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Umbraco.Deploy.Work.WorkItems.SourceDeployWorkItem.<ExecuteAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Umbraco.Deploy.Work.WorkItems.SourceDeployWorkItem.<ExecuteAsync>d__15.MoveNext()

REMOTE:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Umbraco.Web.Cache.DistributedCacheBinder.HandleEvents(IEnumerable`1 events)
   at Umbraco.Deploy.DeployEventDispatcher.ScopeExitCompleted()
   at Umbraco.Core.Events.QueuingEventDispatcherBase.ScopeExit(Boolean completed)
   at Umbraco.Core.Scoping.Scope.<>c__DisplayClass72_0.<RobustExit>b__1()
   at Umbraco.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
   at Umbraco.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
   at Umbraco.Core.Scoping.Scope.RobustExit(Boolean completed, Boolean onException)
   at Umbraco.Core.Scoping.Scope.DisposeLastScope()
   at Umbraco.Core.Scoping.Scope.Dispose()
   at Umbraco.Deploy.Core.AltScope.Dispose()
   at Umbraco.Deploy.Work.WorkItems.SessionWorkItem.<>c__DisplayClass23_0.<Complete>b__0()
   at Umbraco.Deploy.Work.WorkTask.TryDo(Action action)
   at Umbraco.Deploy.Work.WorkContext.TryDo(Action action)
   at Umbraco.Deploy.Work.WorkItems.SessionWorkItem.Complete(Boolean success)
   at Umbraco.Deploy.Work.WorkItems.SessionWorkItem.<ExecuteCompleteAction>b__22_0()
--- End of stack trace from previous location where exception was thrown ---
   at Umbraco.Deploy.Work.WorkItems.SessionWorkItem.WaitIdle()
   at Umbraco.Deploy.Environments.EnvironmentController.WaitCompleted(SessionActionModel model)
   at lambda_method(Closure , Object , Object[] )
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass6_2.<GetExecutor>b__2(Object instance, Object[] methodParameters)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__6.MoveNext()
--> System.NullReferenceException: Object reference not set to an instance of an object.

   at Umbraco.Web.PublishedCache.NuCache.ContentStore.ClearBranchLocked(ContentNode content)
   at Umbraco.Web.PublishedCache.NuCache.ContentStore.SetBranch(Int32 rootContentId, IEnumerable`1 kits)
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.NotifyLocked(IEnumerable`1 payloads, Boolean& draftChanged, Boolean& publishedChanged)
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.Notify(JsonPayload[] payloads, Boolean& draftChanged, Boolean& publishedChanged)
   at Umbraco.Web.Cache.ContentCacheRefresher.Refresh(JsonPayload[] payloads)
   at Umbraco.Core.Sync.ServerMessengerBase.DeliverLocal[TPayload](ICacheRefresher refresher, TPayload[] payload)
   at Umbraco.Core.Sync.ServerMessengerBase.Deliver[TPayload](ICacheRefresher refresher, TPayload[] payload)
   at Umbraco.Core.Sync.ServerMessengerBase.PerformRefresh[TPayload](ICacheRefresher refresher, TPayload[] payload)
   at Umbraco.Web.Cache.DistributedCache.RefreshByPayload[TPayload](Guid refresherGuid, IEnumerable`1 payloads)
   at Umbraco.Web.Cache.DistributedCacheExtensions.RefreshContentCache(DistributedCache dc, TreeChange`1[] changes)
   at Umbraco.Web.Cache.DistributedCacheBinder.ContentService_TreeChanged(IContentService sender, EventArgs args)

To fix the ysod on the remote environment, it seems to work by reloading memory cache in NuCache dashboard, but it doesn't fix the deployment error, and when doing another deploy from local to remove environment the ysod is back.

Furthermore the remote environment shows this log error.

In this case it seems to complain about this loop in line 36 (the code below has been simplified a bit).

@inherits UmbracoViewPage
@{
    var site = Model.Root() as Frontpage;
    var settings = site.FirstChild<Settings>();
    var excludedDoctypes = new[] { Settings.ModelTypeAlias, NotFoundPage.ModelTypeAlias, EmployeeArchive.ModelTypeAlias };

    var menu = site.Children(x => x.IsVisible() && !excludedDoctypes.Contains(x.ContentType.Alias));

    <ul class="list-inline">
        <li class="list-inline-item"><a href="@site.Url">@site.Name</a></li>
        @foreach (var page in menu)
        {
                <li class="list-inline-item"><a href="@page.Url">@page.Name</a></li>
        }
    </ul>
}

[bjarnef]

It seems to be an issue when deploying root node including descedants. I could deploy the root node itself and some other nodes individually, but when I deployed an archive node (listview) with child nodes it failed and shows these log errors on the remote environment. The remote environment however didn't throw an ysod.

In this case it seems it hit this line: https://github.com/umbraco/Umbraco-CMS/blob/3bfd9b71e290354744d677440983ecd06c5c0788/src/Umbraco.Web/PublishedCache/NuCache/PublishedContent.cs#L295

After this I could suddenly deploy this archive node to remote environment and also deploy the root node with descendants.

[Emetico]

Hi Bjarnef.

Its not quite clear to me on how you resolved this, could you explain?

[bjarnef]

@Emetico after the deploy caused an ysod on the remote enviroment, because of a NuCache panic exception, I fixed it on the remove environment by clicking this button to reload memory cache.

It is however not a permanent fix, because the ysod if back on the deploy.

[Emetico]

Thanks alot.

I just have to remember to reload the memory cache each time i deploy until it is fixed.

[bjarnef]

Yeah, it is a bit annoying :)

[nul800sebastiaan]

Hey @bjarnef @Emetico - I'm looking through older issues, can you tell me if this is still an issue with a recent Umbraco + Deploy combination?

[Jay-umbr]

Still an issue on Umbraco 8.8.0 Forms 8.5.3 Deploy 3.5.1

[Jay-umbr]

In which case AppData models are being used. I will see if it can be reproduced on a new install - will update this comment later on.

[umbrabot]

Hiya @bjarnef,

Just wanted to let you know that we noticed that this issue got a bit stale and might not be relevant any more.

We will close this issue for now but we're happy to open it up again if you think it's still relevant (for example: it's a feature request that's not yet implemented, or it's a bug that's not yet been fixed).

To open it this issue up again, you can write @umbrabot still relevant in a new comment as the first line. It would be super helpful for us if on the next line you could let us know why you think it's still relevant.

For example:

@umbrabot still relevant This bug can still be reproduced in version x.y.z

This will reopen the issue in the next few hours.

Thanks, from your friendly Umbraco GitHub bot :robot: :slightly_smiling_face: