The process cannot access the file NuCache.Content.db because it is being used by another process

[robertjf]

PR: https://github.com/umbraco/Umbraco-CMS/pull/5924

Something is happening in Azure WebApps where the NuCache.Content.db file is locked causing the site to hang. I've attached the log file for reference.

The exception is as follows:

Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.

-> Umbraco.Core.Exceptions.BootFailedException: Boot failed.

-> System.IO.IOException: The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\NuCache\NuCache.Content.db' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at CSharpTest.Net.IO.TransactedCompoundFile..ctor(Options options)
   at CSharpTest.Net.Storage.BTreeFileStoreV2..ctor(Options options)
   at CSharpTest.Net.Collections.BPlusTree`2.OptionsV2.CreateStorage()
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheBase..ctor(BPlusTreeOptions`2 options)
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNormal..ctor(BPlusTreeOptions`2 options)
   at CSharpTest.Net.Collections.BPlusTree`2..ctor(BPlusTreeOptions`2 ioptions)
   at Umbraco.Web.PublishedCache.NuCache.DataSource.BTree.GetTree(String filepath, Boolean exists)
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService..ctor(Options options, IMainDom mainDom, IRuntimeState runtime, ServiceContext serviceContext, IPublishedContentTypeFactory publishedContentTypeFactory, IdkMap idkMap, IPublishedSnapshotAccessor publishedSnapshotAccessor, IVariationContextAccessor variationContextAccessor, IUmbracoContextAccessor umbracoContextAccessor, ILogger logger, IScopeProvider scopeProvider, IDocumentRepository documentRepository, IMediaRepository mediaRepository, IMemberRepository memberRepository, IDefaultCultureAccessor defaultCultureAccessor, IDataSource dataSource, IGlobalSettings globalSettings, ISiteDomainHelper siteDomainHelper, IEntityXmlSerializer entitySerializer, IPublishedModelFactory publishedModelFactory, UrlSegmentProviderCollection urlSegmentProviders)

...

This is on Umbraco 8.0.1 which was upgraded from Umbraco 8.0.0. (attachment will need to be changed from .txt to.json)

UmbracoTraceLog.RD2818786B7D96.20190319.txt

[nul800sebastiaan]

This sounds like a configuration problem, at least for @JoseMarcenaro - it might be good to triple-check all differences between those 3 servers.

Other than that, if it can be reproduced with a clean install then we can have a look at additional fixes.

[kjac]

@nul800sebastiaan This also happened to me with 8.1.2 when using deployment slots on Azure. Here are my findings.

Symptoms

Whenever a deployment slot swap was executed, the target (production) slot would crash with the The process cannot access the file error. The source slot would still boot fine when requested after the swap.

This happened with manual swaps from the Azure portal as well as with automated ones from DevOps.

Workarounds

Setting Umbraco.Core.LocalTempStorage = EnvironmentTemp did not help. Only difference was a new file path in the The process cannot access the file error (as one would expect given the new LocalTempStorage setting value).

However! Applying @Shazwazza's fix did the trick. The code looks like this in 8.1.2:

using Umbraco.Core.Composing;
using Umbraco.Web.PublishedCache.NuCache;

namespace Your.Namespace.Here
{
    public class IgnoreLocalDbComposer : IUserComposer
    {
        public void Compose(Composition composition)
        {
            composition.Register(factory => new PublishedSnapshotServiceOptions
            {
                IgnoreLocalDb = true
            });
        }
    }
}

Since we can do warm-ups of deployment slots before swapping, any startup performance impact is of no concern.

[Shazwazza]

My 'fix' will be fine if that's what you choose to use but i recommend reading this whole thread to understand. Setting IgnoreLocalDb = true just means there is no persisted cache file on disk which can affect startup time, but with slot swapping you probably don't need to worry about that either.

What I would like to understand though is what the actual new problem is. It's unfortunate when people say things like "the problem still exists" or "I have the same problem" which isn't entirely helpful since the first problem was definitely fixed and was caused by a race condition. As it seems that there is a new/similar issue then please open up a new issue with steps to replicate and full details. Thanks!

[brreisner]

I can confirm that IgnoreLocalDb = true avoids the issue for us. I will say that we have experienced the issue without slot swapping, even occuring (rarely) when we deploying directly to the primary slot of our dev/qa instances. Will try to identify the issue further but it is difficult to track down.

[Shazwazza]

There's 2 workarounds - one is IgnoreLocalDb, the other is to set WEBSITE_DISABLE_OVERLAPPED_RECYCLING in appSettings to `, see https://github.com/umbraco/Umbraco-CMS/issues/6546#issuecomment-545224926

I will update the hosting on azure docs to include this setting, this is preferable to use over IgnoreLocalDb if you aren't using slot swapping.

[Shazwazza]

docs PR https://github.com/umbraco/UmbracoDocs/pull/2042

[stodolos]

Is there an equivalent for PublishedSnapshotServiceOptions in v8.0.2? We will upgrade to the latest soon, but we have to do some testing for that first.

Update: I just looked at the code, and it looks like you can't. Plus, this version of PublishedSnapshotService does not have the same locking, but please let me know if there is a way to do this in 8.0.2.

Thanks

[deMD]

Time to reopen this issue. I get it on Umbraco 8.4 with the following message:

Exception Details: Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.

-> Umbraco.Core.Exceptions.BootFailedException: Boot failed.

-> System.IO.IOException: The process cannot access the file 'D:\local\Temp\UmbracoData\2d6f8e550e7c1266526ef85adfa0edb1\NuCache\NuCache.Content.db' because it is being used by another process.
  at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
  at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
  at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
  at CSharpTest.Net.IO.TransactedCompoundFile..ctor(Options options)
  at CSharpTest.Net.Storage.BTreeFileStoreV2..ctor(Options options)
  at CSharpTest.Net.Collections.BPlusTree`2.OptionsV2.CreateStorage()
  at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheBase..ctor(BPlusTreeOptions`2 options)
  at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNone..ctor(BPlusTreeOptions`2 options)
  at CSharpTest.Net.Collections.BPlusTree`2..ctor(BPlusTreeOptions`2 ioptions)
  at Umbraco.Web.PublishedCache.NuCache.DataSource.BTree.GetTree(String filepath, Boolean exists) in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\DataSource\BTree.cs:line 27
  at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.MainDomRegister() in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\PublishedSnapshotService.cs:line 175
  at Umbraco.Core.MainDom.Register(Action install, Action release, Int32 weight) in D:\a\1\s\src\Umbraco.Core\MainDom.cs:line 102
  at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService..ctor(PublishedSnapshotServiceOptions options, IMainDom mainDom, IRuntimeState runtime, ServiceContext serviceContext, IPublishedContentTypeFactory publishedContentTypeFactory, IdkMap idkMap, IPublishedSnapshotAccessor publishedSnapshotAccessor, IVariationContextAccessor variationContextAccessor, IProfilingLogger logger, IScopeProvider scopeProvider, IDocumentRepository documentRepository, IMediaRepository mediaRepository, IMemberRepository memberRepository, IDefaultCultureAccessor defaultCultureAccessor, IDataSource dataSource, IGlobalSettings globalSettings, IEntityXmlSerializer entitySerializer, IPublishedModelFactory publishedModelFactory, UrlSegmentProviderCollection urlSegmentProviders) in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\PublishedSnapshotService.cs:line 124
  at DynamicMethod(Object[] )
  at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__2(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__2(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__0(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4646
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.CreateDelegate(Type serviceType, String serviceName, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4743
  at LightInject.ServiceContainer.CreateDefaultDelegate(Type serviceType, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4705
  at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3437
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 111
  at Umbraco.Core.Composing.ComponentCollectionBuilder.CreateItem(IFactory factory, Type itemType) in D:\a\1\s\src\Umbraco.Core\Composing\ComponentCollectionBuilder.cs:line 33
  at Umbraco.Core.Composing.CollectionBuilderBase`3.<>c__DisplayClass10_0.<CreateItems>b__0(Type x) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 100
  at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
  at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
  at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
  at Umbraco.Core.Composing.CollectionBuilderBase`3.CreateItems(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 99
  at Umbraco.Core.Composing.ComponentCollectionBuilder.CreateItems(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\ComponentCollectionBuilder.cs:line 25
  at Umbraco.Core.Composing.CollectionBuilderBase`3.CreateCollection(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 117
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.<>c__DisplayClass20_0`1.<Register>b__0(IServiceFactory f) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 172
  at DynamicMethod(Object[] )
  at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.CreateDelegate(Type serviceType, String serviceName, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4743
  at LightInject.ServiceContainer.CreateDefaultDelegate(Type serviceType, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4705
  at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3437
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 111
  at Umbraco.Core.FactoryExtensions.GetInstance[T](IFactory factory) in D:\a\1\s\src\Umbraco.Core\FactoryExtensions.cs:line 22
  at Umbraco.Core.Runtime.CoreRuntime.Boot(IRegister register, DisposableTimer timer) in D:\a\1\s\src\Umbraco.Core\Runtime\CoreRuntime.cs:line 158

I have the following keys in my appsettings:

    <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
    <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />
    <add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1" />

And I have the workaround still in the application:

public class UmbracoPersistedLocalCacheBugWorkaroundComposer
    {
        public void Compose(Composition composition)
        {
            composition.Register(factory => new PublishedSnapshotServiceOptions { IgnoreLocalDb = true });
        }
    }

I was also able to make it reproduceable for better or worse. The issue always occurs when I manually update the configuration is Azure.

[Shazwazza]

@deMD so to confirm - if you setup a vanilla 8.4 site on azure with

    <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
    <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />
    <add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1" />

you can replicate this problem just by changing an app setting?

[deMD]

@Shazwazza I have not tried it with a vanilla 8.4 yet, the version I have been able to replicate it with was an updated Umbraco install. I'll see what a vanilla instance does.

[kjac]

@deMD I just upgraded a rather complex client site from 8.1.4 to 8.4 in Azure and I can't seem to replicate.

The site has previously experienced this issue. To solve it we added this to the app settings:

<add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.TempEnvDirectoryFactory,Examine" />
<add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />

...and this in a composition:

composition.Register(factory => new PublishedSnapshotServiceOptions
{
    IgnoreLocalDb = true
});

I'm pretty sure the Umbraco.Examine.LuceneDirectoryFactory app setting has nothing to do with the error, but it's included here for good measure - it should be there anyway on Azure.

[JoseMarcenaro]

@deMD, If you want to use the IgnoreLocalDb workaround your class UmbracoPersistedLocalCacheBugWorkaroundComposer should implement IUserComposer as described here in order to be picked up by the initialization pipeline.

[deMD]

@deMD, If you want to use the IgnoreLocalDb workaround your class UmbracoPersistedLocalCacheBugWorkaroundComposer should implement IUserComposer as described here in order to be picked up by the initialization pipeline.

Wow good catch, it seems that indeed our other dev forget that and I completely missed it. Though you would think that workaround shouldn't be necessary anymore, with all the other changes.

@kjac I have the following settings in my app settings:

 <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
    <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />
    <add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1" />

So even without the workaround I would've expected this to work. If the issue is caused by something else (I'll see what an empty clean install of Umbraco does on the same azure resource) then we'll need to dig down into the specific configuration of the Azure environment to see if there is anything there that might cause it.

[JoseMarcenaro]

@deMD you are right - it should work without the IgnoreLocalDB. However, in several Azure / Umbraco 8.3 installations I still got the same error in the past few months so I'm routinely using the workaround.

[deMD]

@Shazwazza I created an empty Umbraco 8.4 website and released it to azure, and I was able to replicate the issue when having added the keys.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.

-> Umbraco.Core.Exceptions.BootFailedException: Boot failed.

-> System.IO.IOException: The process cannot access the file 'D:\local\Temp\UmbracoData\2c21e6f8b472ef713ed2c473e5aee301\NuCache\NuCache.Content.db' because it is being used by another process.
  at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
  at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
  at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
  at CSharpTest.Net.IO.TransactedCompoundFile..ctor(Options options)
  at CSharpTest.Net.Storage.BTreeFileStoreV2..ctor(Options options)
  at CSharpTest.Net.Collections.BPlusTree`2.OptionsV2.CreateStorage()
  at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheBase..ctor(BPlusTreeOptions`2 options)
  at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNone..ctor(BPlusTreeOptions`2 options)
  at CSharpTest.Net.Collections.BPlusTree`2..ctor(BPlusTreeOptions`2 ioptions)
  at Umbraco.Web.PublishedCache.NuCache.DataSource.BTree.GetTree(String filepath, Boolean exists) in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\DataSource\BTree.cs:line 27
  at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.MainDomRegister() in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\PublishedSnapshotService.cs:line 175
  at Umbraco.Core.MainDom.Register(Action install, Action release, Int32 weight) in D:\a\1\s\src\Umbraco.Core\MainDom.cs:line 102
  at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService..ctor(PublishedSnapshotServiceOptions options, IMainDom mainDom, IRuntimeState runtime, ServiceContext serviceContext, IPublishedContentTypeFactory publishedContentTypeFactory, IdkMap idkMap, IPublishedSnapshotAccessor publishedSnapshotAccessor, IVariationContextAccessor variationContextAccessor, IProfilingLogger logger, IScopeProvider scopeProvider, IDocumentRepository documentRepository, IMediaRepository mediaRepository, IMemberRepository memberRepository, IDefaultCultureAccessor defaultCultureAccessor, IDataSource dataSource, IGlobalSettings globalSettings, IEntityXmlSerializer entitySerializer, IPublishedModelFactory publishedModelFactory, UrlSegmentProviderCollection urlSegmentProviders) in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\PublishedSnapshotService.cs:line 124
  at DynamicMethod(Object[] )
  at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__2(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__2(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__0(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4646
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.CreateDelegate(Type serviceType, String serviceName, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4743
  at LightInject.ServiceContainer.CreateDefaultDelegate(Type serviceType, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4705
  at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3437
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 111
  at Umbraco.Core.Composing.ComponentCollectionBuilder.CreateItem(IFactory factory, Type itemType) in D:\a\1\s\src\Umbraco.Core\Composing\ComponentCollectionBuilder.cs:line 33
  at Umbraco.Core.Composing.CollectionBuilderBase`3.<>c__DisplayClass10_0.<CreateItems>b__0(Type x) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 100
  at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
  at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
  at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
  at Umbraco.Core.Composing.CollectionBuilderBase`3.CreateItems(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 99
  at Umbraco.Core.Composing.ComponentCollectionBuilder.CreateItems(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\ComponentCollectionBuilder.cs:line 25
  at Umbraco.Core.Composing.CollectionBuilderBase`3.CreateCollection(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 117
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.<>c__DisplayClass20_0`1.<Register>b__0(IServiceFactory f) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 172
  at DynamicMethod(Object[] )
  at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.CreateDelegate(Type serviceType, String serviceName, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4743
  at LightInject.ServiceContainer.CreateDefaultDelegate(Type serviceType, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4705
  at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3437
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 111
  at Umbraco.Core.FactoryExtensions.GetInstance[T](IFactory factory) in D:\a\1\s\src\Umbraco.Core\FactoryExtensions.cs:line 22
  at Umbraco.Core.Runtime.CoreRuntime.Boot(IRegister register, DisposableTimer timer) in D:\a\1\s\src\Umbraco.Core\Runtime\CoreRuntime.cs:line 158

Here is the ARM template of the azure environment I used: ExportedTemplate-UMBU-REGR-WEU.zip

As for the source I just created an umbraco website without a template, added the keys, and deployed it to Azure.

Then I went into the Azure portal to the Web App, went into the configuration section and added a new app setting and saved. Then it took ~30 seconds to 1 minute for the application to update and crash.

[Shazwazza]

Oh joy, the delights of working with Azure's custom kernels that aren't documented and don't abide by the normal .NET rules. I must have tested this countless times last time we discovered the WEBSITE_DISABLE_OVERLAPPED_RECYCLING setting, looks like I'll need to spend more time re-testing this.

In the meantime, there is some good news on the horizon - I've been working on a new implementation of IMainDom and we've created a new abstraction for this too that can more easily be replaced than the entire MainDom. The PR is here https://github.com/umbraco/Umbraco-CMS/pull/7306 and load testing is still pending on this along with more Azure testing. I'll follow up with more info about this later.

In the meantime, I suggest just using IgnoreLocalDb = true until the new MainDom testing is done and until I have time to re-test why the WEBSITE_DISABLE_OVERLAPPED_RECYCLING is not working again

[vaags]

I can confirm that this is happening in v 8.4 with the keys added. Could this issue be reopened?

[Shazwazza]

As mentioned above we are working on a real solution, not a work around. For now please use IgnoreLocalDb.

I would like to ask if anyone can test if this issue occurs in the latest 8.3.x? - perhaps there's some strange regression in 8.4.

[JoseMarcenaro]

@Shazwazza - not sure about 8.3.0 (I didn't test it) but I did get the error on 8.2.1. As always, the IgnoreLocalDb workaround "fixed" the problem.

[Shazwazza]

I've been testing on Azure with the latest 8.6 codebase. This build includes many of the changes completed and contributed to by @John-Blair which greatly improves the handling of the shutdown sequence along with more robust handling of resource disposal. It also includes a new optional IMainDom implementation that doesn't rely on Semaphores.

But I cannot reproduce this locking issue with this version even with the Semaphore lock!

I have these settings:

• WEBSITE_DISABLE_OVERLAPPED_RECYCLING = 1
• Umbraco.Core.LocalTempStorage = "EnvironmentTemp"
• Umbraco.Examine.LuceneDirectoryFactory = "Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine"

Although i still cannot replicate it even without WEBSITE_DISABLE_OVERLAPPED_RECYCLING.

If you would like to test this out, here's a zip https://umbraco.visualstudio.com/_apis/resources/Containers/7645011?itemPath=zips%2FUmbracoCms.8.6.0-alpha.7394.zip (I'm hoping that zip is available to any microsoft login but let me know if not... it's too big for GH)

I've also tested the other various ways to replicate this issue - togging on/off app insights, adding/removing/changing appSettings via the portal and I cannot get this issue to occur with this build.

To me this indicates that the shutdown changes introduced are very beneficial and we no longer trigger race conditions so we don't see the problem. That said, this doesn't 'fix' the problem because the Semaphore issue is still there and there will be cases where the race condition is triggered.

I've also been testing out the non-semaphore locking mechanism and you can with this build as well by adding this setting:

• Umbraco.Core.MainDom.Lock = "SqlMainDomLock"

This will incur a very simple/fast DB query each second but this shouldn't affect any performance.

If you want to see detailed log outputs for this mechanism add this to your serilog.config file:

<add key="serilog:minimum-level:override:Umbraco.Core.Runtime.MainDom" value="Debug" />
<add key="serilog:minimum-level:override:Umbraco.Core.Runtime.SqlMainDomLock" value="Debug" />

[deMD]

Hey @Shazwazza just came back from vacation so I had no earlier opportunity to check the link out, but sadly it does not appears to be avaiable from my MS login.

[Shazwazza]

@deMD no worries, I've been working on getting to the bottom of this and it turns out that 8.4 has unfortunately included a deadlock issue which I believe is the cause for this.

The issue is here https://github.com/umbraco/Umbraco-CMS/issues/7404

There is a PR now and we'll get an 8.4.1 released with that patched and see how it goes. In the meantime I'm still testing the 8.6 changes with the Sql locking mechanism.

[DanDiplo]

I just encountered this issue on Umbraco 8.4 switching an Azure site from default HTTP/1 to use HTTP/2. Brought down the whole site because the NuCache.Content.db was locked by another process. Just adding this as it might be a useful way of testing this issue.

[Shazwazza]

@DanDiplo yes please try 8.4.1 which contains the #7404 deadlock issue fix https://our.umbraco.com/download/releases/841 also be sure your config is correct.

[danielibanez]

I'm using 8.4.1 and just started getting this issue last night, 02/24/20. Will try upgrading and IgnoreLocalDb = true

Umbraco.Core.Exceptions.BootFailedException: Boot failed.

-> System.IO.IOException: The process cannot access the file D:\home\site\wwwroot\App_Data\TEMP\NuCache\NuCache.Content.db because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

[dalemccutcheon]

Hey @Shazwazza we are still seeing the issue happening on 8.5.2 at the moment. Any suggestions?

[Shazwazza]

@dalemccutcheon i can only really try to help if you have steps to replicate. I don't know your setup but folks tend to miss a configuration step or be doing something outside of the norm that can be part of the problem. Else you can try 8.6 RC (or wait until final release) with this setting https://github.com/umbraco/UmbracoDocs/pull/2330 which is there to be an official working setting for Umbraco on Azure websites.

[danielibanez]

@dalemccutcheon I had this issue pop up on 8.4.1 and 8.6.0-rc. Specifically, I noticed it seemed to happen after switching slots in Azure. But I also had my web.config misconfigured. Umbraco has an Azure guide that, imo, is not very clear:

https://our.umbraco.com/Documentation/Getting-Started/Setup/Server-Setup/azure-web-apps

Anyone who knows otherwise please correct me, but I think as far as configuration goes just make sure these three lines are in your web.config.

<add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp"/> <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine"/> <add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1"/>

I am still on 8.6.0-rc, now with a hopefully properly configured web.config. I will be doing more slot switches and daily log checking.

[Shazwazza]

If you are on 8.6 rc please follow the docs i posted above, here's the link: https://github.com/umbraco/UmbracoDocs/pull/2330, this PR is to update the docs, currently the page will look like this for 8.6 https://github.com/umbraco/UmbracoDocs/blob/4fc755ea1eb67a616e3268c8777fc106a3cb8128/Getting-Started/Setup/Server-Setup/azure-web-apps.md

you need

<add
    key="Umbraco.Core.MainDom.Lock"
    value="SqlMainDomLock" />

[yaplex]

hi, I have the same issue with Umbraco 8.5.3 on Azure and I have the following in web.config

    <!-- Azure hosting settings -->
    <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
    <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />
    <add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1" />
    <!-- End Azure hosting settings -->

Is it any logs or something I can provide to help fix this?

I reading it is because Azure doing something with a website. I have multiple 8.5.x websites, and they are on the same application. The issue happening for all of them at the same time, so this theory seems correct.

[dalemccutcheon]

Hey,

I was still getting the issue however i have added this :

[RuntimeLevel(MinLevel = RuntimeLevel.Run)] public class UmbracoCacheWorkaround : IUserComposer { public void Compose(Composition composition) { composition.Register(factory => new PublishedSnapshotServiceOptions { IgnoreLocalDb = true }); } }

I have not experienced the issue since added the composer.

Cheers Dale

[Shazwazza]

yes the IgnoreLocalDb = true will always work, the caveat is that you will incur a DB performance penalty on startup since the in-memory caches will be rebuilt from the DB instead of the persisted nucache file. This will be fine in most cases but if you have an Umbraco site with tons of content it might not be ideal.

for 8.6 please try https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-599546750

[OwainJ]

@Shazwazza

Apologies for bumping this thread but we're still experiencing this issue on Umbraco 8.8.0, Umbraco.Core.Exceptions.BootFailedException: Boot failed.\n\n-> System.IO.IOException: The process cannot access the file 'C:\\local\\Temp\\UmbracoData\\a30e1f44822dd8ffa5319bc32ef15cb6\\NuCache\\NuCache.Content.db' because it is being used by another process

We have the recommended Azure App Settings in our web.config: <add key="Umbraco.Core.MainDom.Lock" value="SqlMainDomLock" /> <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" /> <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />

Our app is load balanced with the Azure Application Gateway, we have 2 front-end web apps and 1 back-office/master web app with explicit master scheduling set on boot as described here: Explicit Master Scheduling Server Docs

The IgnoreLocalDb = true method works, but as our site has over 250k Media nodes, we can't use that method because it causes our boot time to be around 30 mins. We only removed this today and then got the NuCache lock issue.

Also, is the WEBSITE_DISABLE_OVERLAPPED_RECYCLING setting still required? As this documentation doesn't mention if it is still needed in Umbraco v8.6.4+. We don't currently have it set, but I'll add it and report back if it changes anything.

[JoseMarcenaro]

@OwainJ - just in case, if you do set the WEBSITE_DISABLE_OVERLAPPED_RECYCLING flag, remember to do it on Azure configuartion and not in the web.config - it is ignored there.

[deMD]

Also set the WEBSITE_DISABLE_OVERLAPPED_RECYCLING flag on the Master server if load balancing, even though not mentioned on the documentation we found errors are still caused if not setting that flag on Master environments.

[OwainJ]

@JoseMarcenaro @deMD Thank you both 🙂 I'll keep that in mind.

[stefankip]

We have just encountered this issue as well with v8.8.3 and the recommended Azure settings:

<add key="Umbraco.Core.MainDom.Lock" value="SqlMainDomLock" />
<add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
<add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />

Is there no fix for this yet?

[John-Blair]

I think the SqlMainDomLock was meant to make the following setting unnecessary - but you could give it a try as it worked before the SqlMainDomLock was introduced to fix the problem.

add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1"

The above setting prevents 2 instances of IIS running concurrently - which is what can happen when you make config changes to a web app on Azure.

[stefankip]

@John-Blair thanks, already added that settings right after my previous post. Let's hope it helps.

[danielibanez]

I think the SqlMainDomLock was meant to make the following setting unnecessary - but you could give it a try as it worked before the SqlMainDomLock was introduced to fix the problem.

add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1"

The above setting prevents 2 instances of IIS running concurrently - which is what can happen when you make config changes to a web app on Azure.

This setting needs to be added in Azure and not in your webconfig.

https://our.umbraco.com/documentation/getting-started/setup/server-setup/load-balancing/azure-web-apps

[stefankip]

I know, why did you assume I didn't?

[danielibanez]

This problem seems to be back. I have the settings above as well as

add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1"

in Application Settings in Azure. For those wanting a temporary fix add below to an existing or new composer.

composition.Register(factory => new PublishedSnapshotServiceOptions { IgnoreLocalDb = true });

[stefankip]

This night the same error occurred and the website went down again, although the Azure setting WEBSITE_DISABLE_OVERLAPPED_RECYCLING is set to 1.
Is there really no fix for this besides the work-around of setting IgnoreLocalDb = true which impacts performance?

[danielibanez]

I just had this happen today again and have

IgnoreLocalDb = true

Am completely out of ideas. Any help would be appreciated. cc @Shazwazza

[Shazwazza]

If IgnoreLocalDb == true, then there is no nucache db files in use. Perhaps verify that you have declared this code in the right place and that its executing.

If you your or @stefankip can post steps to replicate the problem with vanilla Umbraco (vLatest) then we might have something to go by. I cannot replicate the issue sorry. The standard azure config should work. Doesn't matter about WEBSITE_DISABLE_OVERLAPPED_RECYCLING (else we would have that in docs).

[enkelmedia]

Hi!

I just want to document my issues reg. this matter on a self hosted IIS-install.

Some background: We use Octopus Deploy for our releases, when it creates a new release it basically creates a new folder and points the IIS Website to this new folder. This means that we can't store any files in the "web root" that needs to be persisted after the new release since this is in a new folder.

Back in the V7-days we solved this by configuring a cache-folder (using web.config appSetting umbracoContentXML) and another folder for the lucene-files, both was configured as virtual directories in IIS so that the same files was used.

Now since our first V8 site we have had this issue from time to time, the Overlaped Recyle-feature is disabled and the appSetting "Umbraco.Core.MainDom.Lock" is set to "SqlMainDomLock" but we still see the problem after about 50% of the releases. It's like some background thread is holding on to the lock of the NuCache-files. I've also tried to stop the App Pool before the release and start it after, and also manual recyle after a release but none of these will work. We either have to wait for the lock to release or restart the entire IIS-web server to release the lock.

Since the path to ie. NuCache is built using Path.Combine(_globalSettings.LocalTempPath, "NuCache"); we have to choose between these issues during deploy OR having to rebuild all the temp files after each release.

A solution that would work temporarily for us with the self hosted scenario would be to make it possible to configure "EACH type of folder" and not just all TEMP-stuff in one folder. So that the NuCache would have it's own override setting, like.

<appSettings>
  <add key="Umbraco.Core.TempStorage.NuCache" value="c:\myfolder\out-side-webroot" />
  <add key="Umbraco.Core.TempStorage.Examine" value="c:\myfolder\another-out-side-webroot" />
  <add key="Umbraco.Core.TempStorage.Logs" value="c:\myfolder\my-logs-folder" />
</appSettings>

Of course the best thing would be to solve the underlying issue that something is holding on to the nucache-files, but it's not reproduceable in a test environment - for us it's only when the site is under load.

@Shazwazza I think that this issue should be reopened and either solved or that we implement the work arounds that I purpose.

Edit: After I managed to configure the Examine-folders using a custom LuceneDirectoryFactory I've managed to get to a state where the site can start up without the locks. The problem is that this means that the new release-folder will not have the NuCache-files so the startup takes around 40-100 seconds - this is better than a YSOD but it would be really nice to get to the bottom off this issue.

@Shazwazza I think that it might be possible to recreate the issue by having two folders with the same files (the website-files, lets call them Folder1 and Folder2) and using the same database etc. Having the IIS-website pointed to Folder1 and start a load test, then during the test switch the IIS website from Folder1 to Folder2. This would mimic what happens when Octopus pushes a release. I also guess that this is a problem when someone does a XCOPY-deploy since the site would restart and the files would be locked by the old process.

In my tests it did not matter if I had the Overlap Recyle feature on or off and the SqlMainDomLock did not matter either. The only "working" solution so far is to recreate the NuCache files in a new folder.

[John-Blair]

Check you have configured Octopus Deploy to shut the existing website down in order to release the locks on the nucache, before pointing the website to the new folder (I understand the same temp files are being used in both) - are you sure it is doing that? Its a bad idea to just recycle the app pool without shutting the website down - this will have random implications for file locks.

[enkelmedia]

We are stopping the app pool as the first step in the process before the release are and starting again as the last step in the process.

[John-Blair]

That sounds like a bad idea to me, and may be the source of your problem. How does just pulling the rug from under the app pool guarantee that all locks on the nucache are released? I would shutdown the website first - this will cause an orderly shutdown of the various caches releasing relevant locks. Only recycle the app pool (if necessary - and it really is not) just prior to starting the website back up.

[enkelmedia]

You did write "configured Octopus Deploy to shut the existing..." so I figured that you referred to the App Pool. What where you referring to? To be clear, we don't Stop/Start the App Pool at the moment but I did test this.

After testing this again now:

Stopping the AppPool before the release-step in the Octopus-process (which is a bad option I know) will actually work, but the site returns "Service Unavailable" for about 30s during the Octopus-step when the site is released. But just adding a recycle of the App Pool as a last step in the process does not work either - the lock's from the old release is not is not released.

As far as I understand, Octopus will basically change the "Physical Path" of the website in IIS to the new folder for the release. Maybe this is the problem that something in the old folder is still "running" when the new release-folder i starting up. For us the lock is not released at all, not even a recycle of the App Pool, only restarting IIS would release the lock.

It's not the MainDom-locks it self as we don't get the System.TimeoutException: Cannot acquire MainDom-exception but just an exception that the NuCache files is in use by another process. This is thrown inside the BTree.GetTree-method.