The process cannot access the file NuCache.Content.db because it is being used by another process

robertjf commented 5 years ago

PR: https://github.com/umbraco/Umbraco-CMS/pull/5924

Something is happening in Azure WebApps where the NuCache.Content.db file is locked causing the site to hang. I've attached the log file for reference.

The exception is as follows:

Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.

-> Umbraco.Core.Exceptions.BootFailedException: Boot failed.

-> System.IO.IOException: The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\NuCache\NuCache.Content.db' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at CSharpTest.Net.IO.TransactedCompoundFile..ctor(Options options)
   at CSharpTest.Net.Storage.BTreeFileStoreV2..ctor(Options options)
   at CSharpTest.Net.Collections.BPlusTree`2.OptionsV2.CreateStorage()
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheBase..ctor(BPlusTreeOptions`2 options)
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNormal..ctor(BPlusTreeOptions`2 options)
   at CSharpTest.Net.Collections.BPlusTree`2..ctor(BPlusTreeOptions`2 ioptions)
   at Umbraco.Web.PublishedCache.NuCache.DataSource.BTree.GetTree(String filepath, Boolean exists)
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService..ctor(Options options, IMainDom mainDom, IRuntimeState runtime, ServiceContext serviceContext, IPublishedContentTypeFactory publishedContentTypeFactory, IdkMap idkMap, IPublishedSnapshotAccessor publishedSnapshotAccessor, IVariationContextAccessor variationContextAccessor, IUmbracoContextAccessor umbracoContextAccessor, ILogger logger, IScopeProvider scopeProvider, IDocumentRepository documentRepository, IMediaRepository mediaRepository, IMemberRepository memberRepository, IDefaultCultureAccessor defaultCultureAccessor, IDataSource dataSource, IGlobalSettings globalSettings, ISiteDomainHelper siteDomainHelper, IEntityXmlSerializer entitySerializer, IPublishedModelFactory publishedModelFactory, UrlSegmentProviderCollection urlSegmentProviders)

...

This is on Umbraco 8.0.1 which was upgraded from Umbraco 8.0.0. (attachment will need to be changed from .txt to.json)

UmbracoTraceLog.RD2818786B7D96.20190319.txt

JoseMarcenaro commented 5 years ago

@zpqrtbnk - would it be possible that one of the composable components we coded in our app, or third-party packages we use (i.e. UmbracoFileSystemProvider.Azure) are delaying the release of the app domain, and causing the collision? That would explain why you cannot reproduce the error.

stodolos commented 5 years ago

We're having the same issue very frequently! Several times a day... Here are the settings we have in our App Service's configuration (not in the config file itself, in case that has any bearing on this):

Umbraco.Examine.LuceneDirectoryFactory = Examine.LuceneEngine.Directories.TempEnvDirectoryFactory, Examine
Umbraco.Core.LocalTempStorage = EnvironmentTemp

Seems similar to this issue regarding Examine locks

Please help!

HannesWinston commented 5 years ago

@stodolos We had this exact problem on two of our Umbraco 8 sites running in Azure Web App. However the issue seems to be fixed now (3 days without a single crash). Here is what we did:

Upgraded Umbraco to 8.0.2
Disabled the Application Insights on the web apps (both production and stage deployment slots)
Completely removed the Application Insights resource in the resource group
Removed all app settings that were automatically applied in the configuration tab on all our web apps (most of them regarding application insights)

Then it worked and no crashes during swaps. I think the 4th step is the most important. See if you have any left over configurations you don't want.

stodolos commented 5 years ago

@HannesWinston Thanks for the info. Once we upgraded to the new version of Examine, we only saw this once in 2 weeks (vs every day). I also saw how 8.0.2 generates the cache file folder/name, and it's way better b/c it goes off of the process id. You would think that should avoid any collision on the same file going forward. Application Insights had no bearing for us FYI.

brreisner commented 5 years ago

Any update on this issue? Does 8.0.2 seem to have resolved the issues for people?

stodolos commented 5 years ago

I still came across this issue in 8.0.2. The folder naming is better (b/c it uses the process ID), but the same process must be trying to access that file more than once. I wonder if something like an epoch date value or a GUID would be possible as to avoid file name collision.

brreisner commented 5 years ago

@zpqrtbnk any details from your side? Should this issue be resolved now or in an upcoming release. Are we even certain on the cause?

brreisner commented 5 years ago

This issue occurred for us with a single node app service plan, no slots, with app insights and appinit enabled. Running on 8.0.2 as well.

This course focuses on running Umbraco 8 load balanced in Azure Web apps and calls out using app insights: https://umbraco.com/blog/introducing-a-brand-new-umbraco-training-course-load-balancing/

Seems like its something that should work. Very concerning, hoping someone can provide some insights as to whether this is something that is being actively worked on, with a fix perhaps by 8.1?

JoseMarcenaro commented 5 years ago

@brreisner - I had a similar situation in a fresh environment I set up this week, and I found a workaround: disabling app preload at the IIS level - yes, I know we are talking about Azure App Services.

The reasoning for trying this is that the conflict is somehow related to the early initialization of the app pool before the previous lock has been released. Then I found this attribute in IIS config <application ... preloadEnabled="true / false" ...> which is by default true in Azure Web Apps.

Setting it to false stopped my NuCache lock conflict from happening on each new deploy / swap.

The thing is: because this is set at the IIS level (applicationHost.config file) you cannot edit it directly on an Azure Web App. But you can modify it by adding a transformation file, as explained here: https://blogs.msdn.microsoft.com/benjaminperkins/2018/01/09/how-to-disable-preloadenabled-on-an-azure-app-service/

NOTES:

If the lock conflict is happening in a multi-slot App Service, you need to repeat this procedure for each of the slots, because they are separate applications on IIS.
If you are using Slot Swap and you are concerned about the warm-up time, you can still use an <applicationInitialization> block in web.config. It does not trigger a NuCache lock conflict, at least in my case.

brreisner commented 5 years ago

@JoseMarcenaro thank you for the update. Has this completely resolved your issues? no further crashes? Have you been provided any official guidance or followup from someone on the Umbraco team. @zpqrtbnk any update here? Will this be addressed in 8.1?

JoseMarcenaro commented 5 years ago

@brreisner

It has completely resolved my issues for now (fingers crossed).
I received no followup about this.

zpqrtbnk commented 5 years ago

We need first, to reproduce and understand. Happy to see that there is a workaround, but we'd love to understand what is really going on. We do have locks in place to ensure that an app restart happens in smooth condition. But the App Preload feature seems to break our locks in very odd ways.

We are trying to prioritize this issue. Will update here.

brreisner commented 5 years ago

Thank you for the follow up as this is concerning for many customers who have already made investments in web apps and have v8 projects moving forward. Are you unable to reproduce still? Seems to be a rather common issue. Jose were you/are you able to provide steps that cause the issue reliably? I would be playing catchup but would be willing to help push this forward.

brreisner commented 5 years ago

Zpqrtbnk please let me know if any of us can help expedite this if you do not yet have the information to properly troubleshoot and isolate the issue. We have several v8 projects in the works scheduled to go live end of October. Do you have any concept of when this might be resolved or when you will be able to accurately describe what is occuring? We just passed 3months since the bug was opened. Not exactly sure what your release cycle looks like for v8, but again I would hope this makes it into 8.1?

azure-devops-sync[bot] commented 5 years ago

This item has been added to our backlog AB#1528

brreisner commented 5 years ago

@Shazwazza Seems as though the backlog is not publicly visible and I'm not sure what that means in terms of priority or timing. Any further insights you can provide?

Shazwazza commented 5 years ago

@brreisner we use a separate proj mgmt tool to help us manage public and internal issues - this is not public. Investigating this has been set to start at the end of this sprint so probably later next week. We'll report back on findings but can't give you an answer because this has never been seen before. Whatever this Azure setting is doing is changing how normal locking at the OS level works.

Shazwazza commented 5 years ago

Hi all, here's a recap - (there's a boat load of information in this thread...)

The problem

When hosting in Azure web apps the locking mechanism in Umbraco that manages overlapping appdomains (called MainDom) is being bypassed in some scenarios which causes file(s) to be accessed by multiple appdomains at one time causing file locking exceptions. In theory this should not be possible because a named Semaphore is a systemwide lock.

These scenarios seem to be:

Using Azure slot swap functionality with the default preloadEnabled IIS setting enabled
Single Azure Web App with Application Insights enabled and with applicationInitialization enabled

Potentially Application Insights may affect this problem but most likely enabling/disabling application insights is probably toggling the preloadEnabled, applicationInitialization or some other IIS feature.

The workarounds

Disabling preloadEnabled IIS setting.
Don't use a persisted local cache file. The content cache is always in memory, the purpose of these persisted files is purely to for startup time performance. That said, the performance of loading in the content cache directly from the DB is also very fast, sometimes negligible. Also considering this is generally a slot swap issue or pre-warming issue, startup time isn't really a problem. To disable persisted file cache you can drop this class in to your app:
- public class MyComposer : IUserComposer { public void Compose(Composition composition) { composition.Register(factory => new PublishedSnapshotService.Options { IgnoreLocalDb = true }); } }

The questions

I need to try to replicate this locally and hopefully the key is in this preloadEnabled setting in IIS. If that is possible then I can try to start figuring out how a named Semaphore stops behaving the way it should be.

Then need to determine if that 2nd scenario listed above (i.e. "Single Azure Web App with Application Insights enabled and with applicationInitialization enabled") can be replicated and then try to determine if the underlying cause is preloadEnabled and/or a combination of applicationInitialization or something else

It would be very helpful if @brreisner and @JoseMarcenaro can verify that both of these workarounds work in your scenarios?

Braindump

If preloadEnabled is confirmed to be the (or part) of the problem, I need to investigate what it's doing. I haven't looked much but there's a bit of info on the web like this which explains:

When you set the startMode property of your application pool to AlwaysRunning a worker process is spawned as soon as IIS starts up and does not wait for the first user request. But this does not mean the web application is initialized. When you set preloadEnabled to true, IIS will simulate a user request to the default page (can be changed with initializationPage metabase setting) of the website/virdir so that the application initializes. The request is not logged in the IIS logs. But you can trace this with FREB. Every time you restart your application pool you will see a FREB trace file for the dummy request. You can identify this request by analysing the GENERAL_REQUEST_HEADERS and looking at the User-Agent string.

which is interesting and if for some crazy reason we cannot figure out why a Semaphore cannot be a Semaphore then we could consider utilizing this information and auto-disabling the persisted file cache.

brreisner commented 5 years ago

I will review these options with my Dev team. I'm a little confused here as I've never (not once) had this issue with v7. This would lead me to believe the logic in v7 is being respected, is something different being done in v8?

Do you have any contacts within Microsoft on the App Service team? I have a few who may be interested in assisting with this.

Shazwazza commented 5 years ago

@brreisner yes this is described above: https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-479942871 . V7 would have it's own issues with this too but they would be hidden from you so you wouldn't know that 2x app domains are writing/reading to the same xml file at the same time until you data is either corrupted or some other exception would cause the site to fail. v7 and v8 are very different :)

I need to replicate locally and gather specific information about what is going on to provide an accurate report before we start involving MS.

In the meantime if you can please try those workarounds that will be very helpful. Both workarounds will still allow slot swap to work and startup time won't 'be an issue anyways.

Shazwazza commented 5 years ago

@brreisner I am trying to replicate the problem you had/have

This issue occurred for us with a single node app service plan, no slots, with app insights and appinit enabled. Running on 8.0.2 as well.

I have configured exactly this with the settings <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />, <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" /> and cannot reproduce.

Can you elaborate on how you reproduce this file lock error with this scenario along with where and when you see the error? The site I'm testing on is https://v8preload.azurewebsites.net/ . I have tried restarting the web app in various different ways, bumping web.config, global.asax, restarting on the portal, changing the /bin contents, exceeding quotas, etc... and I do not see this issue. Any additional info would be appreciated.

Shazwazza commented 5 years ago

Aha! ok well i can replicate the issue once in a while by turning on/off Application Insights (no slot swap involved) though i cannot get this error to happen with a single web app by doing anything else.

I'll test site swapping next to see what I can/can't replicate replicate with/without App Insights.

And locally i cannot replicate with the preloadEnabled, applicationInitialization combos enabled/disabled with overlapping recycling enabled (basically enabling a lot of this article https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-8/iis-80-application-initialization) , but i'll keep testing.

Shazwazza commented 5 years ago

What is different in my tests is that instead of getting this file lock issue, i consistently get the error 502 - Web server received an invalid response while acting as a gateway or proxy server. when toggling Application Insights and when doing a site swap. I've only seen the file lock issue a handful of times and thats mostly in the logs, not as a YSOD. I've seen the YSOD a couple of times during site swap and all of this is with Application Insights fully disabled on both slots.

As mentioned testing locally with the above settings (including preloadEnabled), i cannot break it. I'm load testing and restarting the appdomain concurrently. I've managed to get 5x overlapping concurrent w3wp.exe processes running for the same site and get no file locking issues.

I've discovered a potentially related bug in that when there is overlapping appdomains and when one has been started up and shutdown very quickly, we end up with null reference exceptions in the nucache layer. You can see the rev change I'm making here: https://github.com/umbraco/Umbraco-CMS/commit/3f73bf5489c1a94b4a56c6e52b5b75358a5fdbd7#diff-a7f4263c248ded990a4eac4c9943276f

During testing on Azure, I received this null reference exception in my logs a few times, but i don't see this log entry in the above posted logs so although it might be related i don't think it's a cause.

I've discovered a separate issue with slot swap as well regarding the Application Url so will need to get that fixed. It is related to this pre-warming of a site and Azure behind the scenes makes a request to the site on localhost:80 which means the ApplicationUrl is updated to use this and during pre-warming will send a keep alive request which will fail.

Further testing tomorrow....

JoseMarcenaro commented 5 years ago

@Shazwazza , Workaround # 1 works for us ... most of the times (!?). It fails from time to time (same exception)

Workaround # 2 ... couldn't test it, I don't know how to get past this:

Is there a way to access a class that is not public in Umbraco.Web ? Or do I need to upgrade to 8.1.0 first? (I'm still on 8.0.2)

KraftwerkWedl commented 5 years ago

We are also experiencing the same error: "The process cannot access the file 'D:\local\Temp..."

Running currently Umbraco 8.0.1 on Azure. After deploying to a deployment slot (staging) the application can not start on staging because it runs into that problem, no swap included. After swap, the production runs into same problem. Restart helps to release the lock of course.

I can not add much more for now, what I wanted to test also is what @Shazwazza suggested, to not create persisted files, that would be the ideal solution for us.

But as @JoseMarcenaro mentioned, that class PublishedSnapshotService is internal as far as I see.

Shazwazza commented 5 years ago

Sorry folks, i didn't notice the internal bits, will have to update in 8.1.1. In the meantime, you can use this ugly reflection to disable persisted cache files inside your Compose method:

var optionsType = typeof(IPublishedSnapshotService).Assembly.GetType("Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService").GetNestedType("Options");
var ctor = optionsType.GetConstructor(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic, null, Array.Empty<Type>(), null);
var options = ctor.Invoke(Array.Empty<object>());
var ignoreLocalDbProp = optionsType.GetField("IgnoreLocalDb", BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic);
ignoreLocalDbProp.SetValue(options, true);
composition.Register(optionsType, options);

Shazwazza commented 5 years ago

Good news - the latest changes I've made have resolved the issue. The issue i mentioned here https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-509966706 is the reason why the files get locked, it is a race condition when booting the nucache system when there's multiple appdomains starting up at once. The other good news is that i can confirm the above work around does work so you can use that if you want to for now - but do please test it out and report back!

With this patch, I've tested multiple times:

Site swap = OK
Toggling App Insights = OK
Site swap where one slot has App Insights and not the other = OK

Have made a draft PR here https://github.com/umbraco/Umbraco-CMS/pull/5854

Turns out @zpqrtbnk mentioned this race condition in another document last week, I guess it's good that 2 people discover the same thing separate research ;)

I cannot replicate the problem locally (naturally) no matter how many appdomains i can manage to startup at the same time so it's odd that Azure is so affected by this.

Although this solves the file lock issue, it doesn't explain why the Semaphore lock logging output doesn't match up. I will investigate more of that tomorrow.

JoseMarcenaro commented 5 years ago

@Shazwazza - great news! Just to be sure: the workaround we should test is the IPublishedSnapshotService / ignoreLocalDb code? Thx

Shazwazza commented 5 years ago

@JoseMarcenaro indeed, that would be great.

brreisner commented 5 years ago

Will work with our dev team to put this fix in today and test. This is great news, really appreciate the work here @shazwazza. I know you mentioned the below:

"Don't use a persisted local cache file. The content cache is always in memory, the purpose of these persisted files is purely to for startup time performance. That said, the performance of loading in the content cache directly from the DB is also very fast, sometimes negligible. Also considering this is generally a slot swap issue or pre-warming issue, startup time isn't really a problem."

But can you think of any unintended consequences of making this change? Perhaps scenarios where it would be more impactful than others?

Very much look forward to hearing more on your findings.

JoseMarcenaro commented 5 years ago

@Shazwazza I tested your reflection code to ignoreLocalDb, with total success

I'm running Umbraco v8.0.2

This is the secuence I followed. 1) Re-enabled the IIS preloadEnabled setting, which I had previously set to false (Workaround #1 above) in both slots (production / stage)

2) Performed a swap - with the old code, before applying workaround #2 ==> site broke with the same error "NuCache.Content.db"

3) After stopping / restarting the site, I repeated the swap for further confirmation ==> site broke again.

4) Deployed the new code with the ignoreLocalDb setting and swap ==> site works!

5) Deployed the new code again to the other slot and swapped again ==> site works!

Regarding the performance difference, if there is any it does not affect the production site because we are using the <applicationInitialization> block in web.config, so when the swap is complete the site is up and running.

I will keep this code in place for now, and let you know if it fails at any time. Thanks!

Shazwazza commented 5 years ago

@brreisner regarding

But can you think of any unintended consequences of making this change? Perhaps scenarios where it would be more impactful than others?

If you are using slot swapping with applicationInitialization like @JoseMarcenaro mentions then i don't see unintended consequences of not having a persisted cache file. The only consequence outside of slot swapping is potentially slower startup but this depends largely on how much content you have. For a site with a couple thousand nodes the startup time would be probably be negligible anyways.

@JoseMarcenaro great news! In 8.1.1 I'll make these options to have a public API so the reflection code will probably stop working so just a heads up! That said, it will be easier to set the options but also 8.1.1 will have the race condition fixed anyways so you prob don't need to worry about anything anyways.

JoseMarcenaro commented 5 years ago

@Shazwazza , thanks for the heads up. I'm eager to test how it works in 8.1.1, meanwhile my problem is gone for now :-)

brreisner commented 5 years ago

@Shazwazza this has fixed our issue as well. Can you clarify on what the ultimate fix is that will be in 8.1.1 and is there any rough time frames on when this will be available? Will we be able to have a persisted cache file once the fix is in? If we can't, does this prevent the use of caching as detailed here: https://our.umbraco.com/documentation/Reference/Cache/?

You mentioned also the issue is present in v7 but not detected in the same manner. Are there fixes that will make there way into v7 or anything we need to do for v7 deployments?

Thanks for all your work to get this addressed!

Shazwazza commented 5 years ago

The 'ultimate fix' ( 😂 ) is that a race condition is fixed so that an appdomain that is started and immediately shutdown because another appdomain comes online moments later does not try to access the nucache persisted files.

Will we be able to have a persisted cache file once the fix is in

yes

If we can't, does this prevent the use of caching as detailed here:

This doesn't prevent you from doing anything. As mentioned before - the persisted cache files are purely there for startup performance times. All cache is in memory regardless of whether it loads from the DB on startup of the perstisted cache files. The usage of Umbraco is identical.

You mentioned also the issue is present in v7 but not detected in the same manner. Are there fixes that will make there way into v7 or anything we need to do for v7 deployments?

This is not in the same context as this issue. Like i said, v7 and v8 are very different. The XML cache in v7 has it's own issues which is one reason why it doesn't exist in v8. These issues have always been there - for example, there is a reason why the right click context menu in the very root 'content' node in the tree has a "refresh all" which doesn't exist in v8. It's because the v7 xml cache in some cases can be "corrupted" due to various different reasons ranging from the file not being exclusively locked and allowing multiple appdomains to read/write to it at the same time. These issues cannot be fixed in v7 - a big reason why v8 exists is so that we could fix these issues.

Shazwazza commented 5 years ago

The next tests that i would love for @JoseMarcenaro and @brreisner to run is to try a nightly of the 8.1.1 Umbraco.Core/Umbraco.Web DLLs. I will upload these here shortly. We would like to release an 8.1.1 asap (probably in a couple weeks) but I cannot close this issue until i know it works for you guys.

DLLs coming soon....

JoseMarcenaro commented 5 years ago

@Shazwazza sure, just post them here and we will test them ASAP.

brreisner commented 5 years ago

Will do, Thanks!

Shazwazza commented 5 years ago

Please see attached the 8.1.1 binaries zip file.

NOTE These are NOT the final release binaries of 8.1.1 and they don't have a pre-release alias on the build. This build was made from rev 36030904212d321af75f96bfc894742e86629ec3

To test:

Take a backup of Umbraco.Core.dll and Umbraco.Web.dll and then replace these 2 DLLs with the ones in the zip file
Remove the work around(s)
- On another note - the reflection work around will no longer work with these DLLs since you don't need to use reflection anymore since we've made this class public: https://github.com/umbraco/Umbraco-CMS/pull/5924/files#diff-8ebcd4b5d06971ef8152c6628b5c9a6aR6 . The work around will still work but you don't need to use ugly reflection to do it
Change your version in your web.config to 8.1.1 instead of 8.1.0 - or you can let the upgrader run
Check if everything works

To rollback:

replace these 2 DLLs with your backup copies
change the version in your web.config back to 8.1.0
re-instate work around

UmbracoCms.AllBinaries.8.1.1.zip

Please let me know how you go, we won't be shipping this release without knowing it works for you.

JoseMarcenaro commented 5 years ago

@Shazwazza - quick question We should do this on top of an existing 8.1.0 installation, right?

The problem I have is that my production site is still 8.0.2, and I cannot upgrade it to 8.1.0 until I get issue #5886 solved (Variant Children of non-variant node are not listed) because it breaks the site in multiple places.

Shazwazza commented 5 years ago

@JoseMarcenaro yeah, let me make a build of 8.0.x with this fix for you to test and i'll also look into the children issue.

Shazwazza commented 5 years ago

@JoseMarcenaro here's an 8.0.2 build with the fix in these 2x DLLs. FYI these are Debug versions of these DLLs (not Release). I've just built this in VS based on the 8.0.2 tag and cherry picked the commit rev 36030904212d321af75f96bfc894742e86629ec3 ... i think that should work for you, please let me know

8.0.2_CoreAndWeb.zip

JoseMarcenaro commented 5 years ago

@Shazwazza thanks for building this DLL set for me.

Good news: On an existing 8.0.2 Azure web app, built without the reflection code, I dropped these DLLs in both slots (stage - production) and swapped back and forth without issues (no NuCache blocking)

Then I completed the setup of a 8.1.0 separate environment, and dropped the other DLL zip you provided for 8.1.1 (in both slots) and swapped back and forth - no incidents! Yay! Of course, on the first run it upgraded the DB to 8.1.1 as expected.

It looks like the fix in the initialization code solves the lock issues. Great work! Thanks, Jose

Shazwazza commented 5 years ago

@JoseMarcenaro many thanks for testing! I'll go ahead and close this issue. @brreisner will still be great if you can let me know the outcome of your testing too. Cheers!

warrenbuckley commented 5 years ago

Closing this issue as this got merged in from this PR https://github.com/umbraco/Umbraco-CMS/pull/5924

ealse commented 5 years ago

I have updated Umbraco from version 8.1.0 -> 8.1.1

When I was using version 8.1.0 with reflection to disable persisted cache files inside my Compose method (As mentioned above) no errors were thrown.

After updating to version 8.1.1 and removing the workaround the following error came back when swapping my web app: Umbraco.Core.Exceptions.BootFailedException: Boot failed. -> System.IO.IOException: The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\NuCache\NuCache.Content.db' because it is being used by another process

UmbracoTraceLog.txt

jsheard02 commented 5 years ago

@ealse @Shazwazza

We have tried this on 8.1.0, 8.1.1 and 8.1.2, and we experience the same NuCache.Content.Db error as you point out. For some additional context, we add the umbracoLocalTempStorage into the Application Settings of the Web App on azure, and not directly into the web app itself. I will try and add this directly into the web.config file now, and let you know if the issue persists.

Shazwazza commented 5 years ago

@jsheard02 umbracoLocalTempStorage isn't the correct setting, that is for v7, see https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-476477802

anthonydotnet commented 5 years ago

This doesnt seem to work in v8.1.2. We still need to restart the app after a slot swap :(

JoseMarcenaro commented 5 years ago

We have three similar Azure environments with v8.1.1 In one of them we get the error every time - the other two are good. And I can't figure out what the difference is. Any suggestions for further testing?

umbraco / Umbraco-CMS