Azure App Service Deploy fails for .net core 2.0 with ERROR_FILE_IN_USE

[IvanAlekseev]

I am trying to deploy an Asp.Net Core 2.0 web app to Azure App Service and receive this error

2017-09-07T15:20:07.7379097Z ##[section]Starting: Deploy Azure App Service
2017-09-07T15:20:07.7569101Z ==============================================================================
2017-09-07T15:20:07.7569101Z Task         : Azure App Service Deploy
2017-09-07T15:20:07.7569101Z Description  : Update Azure Web App Services, Web App On Linux , Function Apps, Mobile Apps using Web Deploy / Kudu REST APIs
2017-09-07T15:20:07.7569101Z Version      : 3.3.14
2017-09-07T15:20:07.7569101Z Author       : Microsoft Corporation
2017-09-07T15:20:07.7569101Z Help         : [More Information](https://aka.ms/azurermwebdeployreadme)
2017-09-07T15:20:07.7569101Z ==============================================================================
2017-09-07T15:20:11.9902809Z Got connection details for Azure App Service:'██████████████'
2017-09-07T15:20:16.5780995Z [command]"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package='d:\a\r1\a\██████████████\drop\██████████████.zip' -dest:auto,ComputerName='https://██████████████.scm.azurewebsites.net:443/msdeploy.axd?site=██████████████',UserName='********',Password='********',AuthType='Basic' -setParam:name='IIS Web Application Name',value='██████████████' -enableRule:AppOffline -retryInterval:6000 -retryAttempts:10 -enableRule:DoNotDeleteRule -userAgent:VSTS_e29e8f6a-6b5e-463d-ba6e-72dd7866010e_release_10_50_99_1
2017-09-07T15:20:17.1910281Z Info: Using ID '████████████████████████████████████' for connections to the remote server.
2017-09-07T15:20:21.1058805Z Info: Using ID '████████████████████████████████████' for connections to the remote server.
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\appsettings.Development.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\appsettings.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\Connected Services\Application Insights\ConnectedService.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.Contract.dll).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.Contract.pdb).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.deps.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.dll).
2017-09-07T15:21:24.7852633Z ##[error]Failed to deploy web package to App Service.
2017-09-07T15:21:24.7862628Z ##[warning]Try to deploy app service again with Rename locked files option selected.
2017-09-07T15:21:24.7862628Z ##[error]Error Code: ERROR_FILE_IN_USE
More Information: Web Deploy cannot modify the file '██████████████.dll' on the destination because it is locked by an external process.  In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.
Error count: 1.

2017-09-07T15:21:24.7862628Z ##[error]Error: C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe failed with return code: 4294967295
2017-09-07T15:21:26.2210992Z Successfully updated deployment History at https://██████████████.scm.azurewebsites.net/deployments/501504797684782
2017-09-07T15:21:26.2321177Z ##[section]Finishing: Deploy Azure App Service

I've tried some workarounds from similar issue without any luck https://github.com/Microsoft/vsts-tasks/issues/1607

Any ideas on what should I do to fix this?

[niadak]

@frankfuu , Did you get a chance to try the above suggestion? Are you still facing the issue?

[frankfuu]

hi @niadak Yep, it seems to be working so far! I'm hoping to kind of go back and see how many times it actually did retry.

[DaleMckeown]

It should be noted that this issue is not limited to Azure - publishing to IIS via Web Deploy in VS causes the same issues.

[buzallen]

This is still an issue for me, but only on certain projects and I can't find a common thread as of this point. This is a .net core 2.0 MVC application being published to an azure web app from visual studio using the publishing profile. Only way around it is to stop the service and then restart it. Note this is not a slot in this case, it's the regular production app.

Over the summer this was an issue on some other projects but that went away and I assumed the problem was fixed. In two new projects this issue is back. I have the retry attempts set at 20 but per another issue thread that is not needed anymore.

I'd guess about 50% of publishes fail with this error. I know I can use a staging slot and swap to make it work and on our primary app I do that but there are some smaller projects I'd really like to just be able to publish simply.

[vincent1173]

@buzallen , Can you please provide the details asked by @pan-wang , so we can closely debug this issue?

[buzallen]

@vincentdass I read the @pan-wang post about event logs, is it the eventlog.xml file in the 'Logs' directory? I've enabled web server logging for the app but the eventlog.xml file doesn't appear to have any errors in it even though a publish failed after I enabled logging.

If there are any details on how to capture the specific logs, or if I am not getting the correct data please let me know. FWIW, I was able to publish without error twice before it failed the third time.

[pan-wang]

@buzallen ANCM logs failure to Windows Event log. You can try the method posted at http://www.c-sharpcorner.com/article/azure-app-service-troubleshoot-using-event-log/ to get the windows event log.

[pgmolloy]

Getting this issue today, I had it a while back. Have to do the same thing stop the service, deploy and restart. Here's the info.

2017-12-31 4:44:09 PM System.AggregateException: One or more errors occurred. ---> System.Exception: Publish failed due to build errors. Check the error list for more details. --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at Microsoft.VisualStudio.Web.Publish.PublishService.VsWebProjectPublish.<>c__DisplayClass40_0.b2() at System.Threading.Tasks.Task`1.InnerInvoke() at System.Threading.Tasks.Task.Execute() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.ApplicationCapabilities.Publish.ViewModel.ProfileSelectorViewModel.d108.MoveNext() ---> (Inner Exception #0) System.Exception: Publish failed due to build errors. Check the error list for more details.<---

===================

Severity Code Description Project File Line Suppression State Error Web deployment task failed. (Web Deploy cannot modify the file 'rsocapiservice.dll' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.) 0

[vincent1173]

@pgmolloy , the recommendation from App Service team is to stop, deploy and start the app (as you stated), when the app_offline doesnt have any effect (even with retry-attempts and retry-interval).

[nordvall]

I've been doing almost a hundred deployments over the last months and never hit this issue. I am deploying Azure Functions and ASP.NET Core 2.0 apps with VSTS Release Management.

However today this issue appeared on one of my apps (ERROR_FILE_IN_USE for my application dll). One special thing about this app is that it was manually stopped from the Azure Portal during the Christmas holidays. Yesterday I did a new deployment, and then manually started the app again from the Azure Portal (when I realized it was stopped). Everything was then working again, but today the deployments starts failing. I have no special parameters configured in the deployment task (since I have not needed any tweaks).

I have no idea about the inner workings of this, but I am thinking that maybe manually starting and stopping apps in some way interfere with the web deployment's ability to release file locks?

I've now surrounded the deployment task with "Azure App Service Manage" start/stop tasks in the process and the deployments now works again.

[vincent1173]

For FILE_IN_USE issue, There are few recommendation that resolves the issue.

• Enable Take App Offline option. With -retryAttempts:10 -retryInterval:6000 additional argument provides a reliability with retry.
• Enable Rename locked files option to allow MSDeploy to attempt to rename DLLs if they can't be copied during deployment The above two options works for most of the cases.

If the issue is still not resolved, try stopping the app service, deploy and then start the app service. Swap slot helps in Zero-down time deployment.

Refer here for more details.

Rever back if the recommendation is not working.

[vincent1173]

Closing the thread as the recommendation provided above should have resolved the issue. Feel free to re-open the thread or new one if the issue is still persisting.

[onionhammer]

This is definitely a hack/workaround, not a very satisfying solution.

[DaleMckeown]

This also doesn't cater for people deploying to a regular IIS site, as reported above.

Stopping an app pool, publishing a service, then restarting an app pool is not a good process for people who have to publish an app to a single server.

[rmarinho]

Please reopen, stopping the website is not what one should be recommend to do.. We have production apps depending on 100% uptime.

[vincent1173]

Reopening the thread. @onionhammer , @rmarinho , Is App_offline and Rename locked files option not resolving your issue? @DaleMckeown , Are you referring to IIS App Deploy task?

[onionhammer]

@vincentdass It works around the core issue. A workaround is a temporary solution to a more fundamental problem. The fundamental problem is by default this task fails without a lot of monkeying around with the settings. Do you expect users in the future to simply search for a solution, find this post, and apply your workaround whenever people come across this in the future?

[vincent1173]

@onionhammer , I agree with your concern. We are currently working on an experience to enable the app offline feature and rename locked files options by default with retry attempts for MSDeploy in the release templates. Need to think through on enabling these options by default in the task, as App offline flag brings the app down for a time-interval and not accepted by others.

Now on the core problem, where the task failing without retrying with the above two options. We explicitly want to point out what went wrong and the recommendation is provided in the error logs as warning message.

2017-09-07T15:21:24.7862628Z ##[warning]Try to deploy app service again with Rename locked files option selected.

More Information: Web Deploy cannot modify the file '██████████████.dll' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt.

[DaleMckeown]

@vincentdass Yep.

[hakankaraduman]

I'm having the same problem on my .net core 2 application. It was okay yesterday but today I have to first stop the app service and then I can upload and restart the app service.

What's the workaround for this? I publish from the visual studio 2017 community edition.

[jonathaneckman]

I got stuck behind the same issue deploying my .NET Core 2 app to an app service using every variety of work around in this thread.

Make sure you queue a new release if you change your release task using the suggestions above. Using redeploy will not pick up changes to your task.

[PaitoAnderson]

I wonder if we could use the new Run-From-Zip deployment option to solve these problems?

Azure/app-service-announcements#84

[davidebbo]

@PaitoAnderson it's not quite ready for prime time, but I do think it will provide a solution to many of these issues.

[Korporal]

@davidebbo @IvanAlekseev

VS 2017 Enterprise, 15.5.6.

I've just encountered this problem after my first experiments in publishing .Net Core Web Apps to Azure. The problem is intermittent for me. I published a simple app this morning and (other than two frustrating but unrelated problems https://github.com/aspnet/JavaScriptServices/issues/1330 and https://github.com/aspnet/JavaScriptServices/issues/1282) had no problem. I published the app several times over the course of an hour as I made tiny tweaks to it (cosmetic).

I then went out for an hour, came back made another small tweak and tried to publish whereupon I hit this locked assembly issue.

I stopped the app in azure and it then published fine, I made a small change and published without stopping it and that too went fine.

I have to say, the presence of this problem strikes me as rather incredible given the fundamental nature of what we're discussing here. If publishing to Azure using the latest MS tools fails like this then what is going on at Microsoft? This issue seems to have been around for quite a while and it's not being treated as a priority 1 bug?

I have no idea what goes on under the hood with a publish, but surely part of the protocol must involve a request to stop the app as one of the first steps?

I'm interested to know too if the problem is in VS 2017 or Azure, does anyone at least know this much?

Is this officially regarded as a bug by Microsoft?

Thanks.

[pan-wang]

I recently got a local repro with VS publishing(not on azure). What I noticed is that the published was trying to delete the whole app folder which is under file watching by ANCM for app_offline change and IIS for configuration change. All files in the target folder is marked as deleting and publishing failed. Interesting thing is that process monitor did not report any other process tried to access files except the publisher. I coulf manually copy and delete files/folders for the same scenario. Will work with VS publisher team to see how we can fix it.

[vincent1173]

@pan-wang , any updates on the issue?

[pan-wang]

@vincentdass Webdeploy made some change to avoid race condition due to file watcher. I have not try the new bits to see whether it also solve our problem. Will do it.

[mdsharpe]

Frequently getting this problem deploying an asp.net core 2.0 web app to Azure using Octopus Deploy. Often have to manually retry many times. Seems intermittent.

[BowserKingKoopa]

I'm having the same problem. I'm using the Publish option in Visual Studio 2017 for an ASP.NET Core on Core app. This has never happened to me before during my years of clicking Publish for classic ASP.NET apps. For my ASP.NET Core apps this happens intermittently, but very frequently. I can get around it by shutting off the app from Azure Portal. This needs a real fix, Publish is broken.

[pan-wang]

@BowserKingKoopa did you enable app-offline for deployment? as there is no shallow copy for core app, appoffline must be used to shutdown the kestrel application. if possible, could you please collect and share the windows event log during deployment? this is log will tell whether the file change notification was received and what the response received from backend for shutting down.

[BowserKingKoopa]

Yes, I've set EnableMSDeployAppOffline to true in the publish profile and it had no effect whatsoever.

[Korporal]

This is hitting people pretty frequently so how about Microsoft making an effort to reproduce it and doing the right thing with this? its close to a "show stopper" for some of us.

Don't lose sight of the fact that MS are competing with the likes of AWS too...

[DaleMckeown]

@Korporal Agreed.

It's getting to the point where I'm actively putting off publishing updates to sites for my organisation now.

Overall, I love AspNetCore, but publishing sites has been a significant step backwards in my opinion.

[PaitoAnderson]

I've stopped using MSDeploy with AspNetCore because of this issue.

I tried Run-From-Zip (Preview), but there are AspNetCore issues with it as well currently (Documented here: Azure/app-service-announcements-discussions#32)

I ended up using ZipDeploy, but I have to manually click "Restart" on the App Service after every deployment to get it to pick up the changes.

EDIT: I love AspNetCore + Azure overall too... it's just this issue that has me down :)

[davidebbo]

@PaitoAnderson (and others) I have some potentially good news for things to come in the next week or so:

• The Run-From-Zip issue is being addressed. The fix is already in some places but not yet everywhere,
• We're making changes to ZipDeploy such that if the app is set up for Run-From-Zip (using WEBSITE_RUN_FROM_ZIP=1, not the external URL), ZipDeploy changes its behavior to use the passed in zip file for Run-From-Zip (instead of just expanding it to wwwroot is it normally would - wwwroot is read-only in this case).

I know this sounds a little complicated, but bottom line is that in theory, to get things working with .NET Core and VSTS, all you have to do is:

• WEBSITE_RUN_FROM_ZIP=1
• Tell vsts to use zipdeploy (and not use app_offline)

Does that make sense?

[pan-wang]

@PaitoAnderson I got a repro. What I found is that for some version VS, ANCM did not get the file change notification when app_offline.htm file was deployed by webdeploy. If manually dropping app_offline.htm, the deployment works. And Unfortunately, webdeploy never tried to redeploy the app_offline.htm during retry. I'm working with WebDeploy developer to see whether we can redeploy the app_offline.htm during retry and may be change ANCM to periodically scan the app_offline.htm file instead of purely relying on file change notification. Will update the thread about the progress.

[PaitoAnderson]

@davidebbo @pan-wang Thank you both for your work and responses on this issue!

I'll try adding WEBSITE_USE_ZIP=1 and report back on my findings next Monday; I'm using East US.

[PaitoAnderson]

I tried added WEBSITE_USE_ZIP=1 on two different App Services today both in East US, (1) Is targeting netcoreapp2.0 using AspNetCore, no problems, zip deployment worked fine too! (2) Is targeting net471 using AspNetCore, instant 500 errors when applying the application setting, any ideas?

[davidebbo]

@PaitoAnderson unfortunately, there were some delays, and the deployment in East US is only partial. To confirm that it is the reason, please try the following:

• For each of the two apps, go to Kudu (AppName.scm.azurewebsites.net)
• Look at the 'Azure App Service' field (not the 'Build'). 71.0.x is the old build, and 72.0.x is the new one

Does that match your working / non-working app?

[PaitoAnderson]

Yes, that matches the working / non-working, I'll try again when that updates. Thank you!

[davidebbo]

@PaitoAnderson Chances are you got the update now, so please give it a try when you have a chance.

[PaitoAnderson]

@davidebbo I tried my best, but I hit different issues, maybe it's me this time.

Zip Deployment: I set WEBSITE_USE_ZIP=1 and I didn't get 500 errors, cool! Next, I did a zip deployment but it didn't pickup my updated dlls (C# changes), html/javascript were new however. Update: This worked the second time, maybe it is fixed!

Run-From-Zip: I deployed to a Azure Blob, got the SAS URL (I tested downloading the zip using a browser), I set WEBSITE_USE_ZIP to the SAS URL but it just started returning 403 "You do not have permission to view this directory" errors.

[davidebbo]

@PaitoAnderson can you email me details (david.ebbo (at) microsoft.com), including the test site name and the zip url? We'll figure out what's going on.

[andreujuanc]

I have an Azure function deployed from VS for testing purposes and I'm having the same issue. And I cant use the renaming flag because i need to update the locked file to a new version.

It's so meh to go to the portal and stop > publish > start every time i publish.

I'm just testing a solution via VS publish profile, havent configure this solution to our CD pipeline.

Any tips will be appreciate it. PS: already have <EnableMsDeployAppOffline>true</EnableMsDeployAppOffline> C:\Program Files\dotnet\sdk\2.1.104\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : Web deployment task failed. (Web Deploy cannot modify the file 'DurableTask.AzureStorage.dll' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.) [C:\Users\andre\Source\Repos\WgmCodeJobs\DurableFunctions\WgmCodeJobs\WgmCodeJobs.csproj]

[jsheetzati]

@davidebbo What is the recommended way to "Tell vsts to use zipdeploy (and not use app_offline)"? Is there a way to do this with the Azure App Service Deploy task?

[davidebbo]

@jsheetzati I'm on the App Service side of things, so not an expert on VSTS. @vincentdass can answer this. Last I heard, they were in the process of adding zipdeploy support, but not sure the status.

[SumiranAgg]

@jsheetzati VSTS consuming zipdeploy is work in progress and we will let you know once it is ready.

[thomasrousset]

Hi,

so to use the fix we have to add to the appsettings' webapps this key WEBSITE_USE_ZIP=1 ; that's it ?

[davidebbo]

@thomasrousset You also need to you zipdeploy for deployment. See my previous comment. Also, note that WEBSITE_USE_ZIP was renamed to WEBSITE_RUN_FROM_ZIP.

[PrimeHydra]

Was hitting this issue trying to deploy a .NET Core 2.0 Azure Function. Tried WEBSITE_USE_ZIP=1, didn't help. Tried checking Take App Offline, no joy.

What worked for me: Checking Publish Using Web Deploy and Rename Locked Files, then creating a new deployment. (Like jonathaneckman said, Redeploy won't work because it won't pick up your changes to the release definition.)