Azure App Service Deploy fails for .net core 2.0 with ERROR_FILE_IN_USE

[IvanAlekseev]

I am trying to deploy an Asp.Net Core 2.0 web app to Azure App Service and receive this error

2017-09-07T15:20:07.7379097Z ##[section]Starting: Deploy Azure App Service
2017-09-07T15:20:07.7569101Z ==============================================================================
2017-09-07T15:20:07.7569101Z Task         : Azure App Service Deploy
2017-09-07T15:20:07.7569101Z Description  : Update Azure Web App Services, Web App On Linux , Function Apps, Mobile Apps using Web Deploy / Kudu REST APIs
2017-09-07T15:20:07.7569101Z Version      : 3.3.14
2017-09-07T15:20:07.7569101Z Author       : Microsoft Corporation
2017-09-07T15:20:07.7569101Z Help         : [More Information](https://aka.ms/azurermwebdeployreadme)
2017-09-07T15:20:07.7569101Z ==============================================================================
2017-09-07T15:20:11.9902809Z Got connection details for Azure App Service:'██████████████'
2017-09-07T15:20:16.5780995Z [command]"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package='d:\a\r1\a\██████████████\drop\██████████████.zip' -dest:auto,ComputerName='https://██████████████.scm.azurewebsites.net:443/msdeploy.axd?site=██████████████',UserName='********',Password='********',AuthType='Basic' -setParam:name='IIS Web Application Name',value='██████████████' -enableRule:AppOffline -retryInterval:6000 -retryAttempts:10 -enableRule:DoNotDeleteRule -userAgent:VSTS_e29e8f6a-6b5e-463d-ba6e-72dd7866010e_release_10_50_99_1
2017-09-07T15:20:17.1910281Z Info: Using ID '████████████████████████████████████' for connections to the remote server.
2017-09-07T15:20:21.1058805Z Info: Using ID '████████████████████████████████████' for connections to the remote server.
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\appsettings.Development.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\appsettings.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\Connected Services\Application Insights\ConnectedService.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.Contract.dll).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.Contract.pdb).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.deps.json).
2017-09-07T15:20:27.0283815Z Info: Updating file (██████████████\██████████████.dll).
2017-09-07T15:21:24.7852633Z ##[error]Failed to deploy web package to App Service.
2017-09-07T15:21:24.7862628Z ##[warning]Try to deploy app service again with Rename locked files option selected.
2017-09-07T15:21:24.7862628Z ##[error]Error Code: ERROR_FILE_IN_USE
More Information: Web Deploy cannot modify the file '██████████████.dll' on the destination because it is locked by an external process.  In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.
Error count: 1.

2017-09-07T15:21:24.7862628Z ##[error]Error: C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe failed with return code: 4294967295
2017-09-07T15:21:26.2210992Z Successfully updated deployment History at https://██████████████.scm.azurewebsites.net/deployments/501504797684782
2017-09-07T15:21:26.2321177Z ##[section]Finishing: Deploy Azure App Service

I've tried some workarounds from similar issue without any luck https://github.com/Microsoft/vsts-tasks/issues/1607

Any ideas on what should I do to fix this?

[vincent1173]

@IvanAlekseev , Can you please share the full logs for the release?

[IvanAlekseev]

@vincentdass added more info. Will it be enough?

[vincent1173]

@IvanAlekseev , Is there any web jobs enabled in App Service (including Application Insights)? Did you enable Rename locked files option too? Please share your app service name without reporting it explicitly

[IvanAlekseev]

@vincentdass - this is a dummy service name - http://errorfileinuse-msexample.azurewebsites.net and we have a problem with recommendations api. We have app insights enabled and no other webjobs. I will try with Rename locked files option and tell you the results, for now we deploy to staging and then swap with prod

[nphmuller]

Same issue here: http://dummy-fileinuse.azurewebsites.net/

Happens on both the staging and production variant of the api app. I have Rename locked files and Take App Offline enabled. Still get the error. Retrying the deployment usually works. Application Insights is enabled in our application. Not as an App Service extensions. No Web Jobs either.

I have tried -retryAttempts:10 but not -retryInterval:6000. Will probably work as a workaround when I try the deployment with both.

[pan-wang]

could you please collect event log from AppService portal and share it? In general, a windows event will be logged once the file change of app_offline.htm is received and the IIS component try to gracefully shutdown the backend asp.net process.

[jeffputz]

There are at least three versions of this issue floating around, all with workarounds that don't work for everyone. It became an issue again with .NET Core v2 release time... perhaps the thing that changed has something to do with it? Having to change your build process and endure a lot of pain with every release isn't great for productivity.

[rmarinho]

Ok let's just talk here :) i have the same issue, workaround is manual turn off the slot.. i don't like that kills productivity ..

[hkusulja]

It worked for me before, and it stopped working about 2 months ago, and still have issues. My app is asp.net core v1 including the asp.net 4.6.1 , I did not change my app or deployment procedure. Issue still persist. Second build always passes. I have same issue for multiple VS TS projects / and azure web sites.

[vincent1173]

@hkusulja , Is App offline flag enabled? Can you please try adding -retryAttempts:10 -retryInterval:6000 in Additional arguments of Web Deploy? In this way, we increase the retry time for deployment.

[jeffputz]

I did that and it had no effect.

[TroySchmidt]

Having the same issue. Using a continuous deployment release definition in VSTS to AWS instance running IIS 8.5.9600.x + Windows 2012 R2 + .NET Core 1.1.2 x64. It randomly fails deploying because the file is still in use. I have tried the retryAttemps, retryInterval, put stop and start management tasks around the deploy. This is still happening regardless of any workarounds other than manually redeploy until it works.

[tomkerkhove]

Same issue here

[ColinM9991-zz]

Having the same issue on an Azure App Service environment with no fix on the retryAttempts and retryInterval.

It works fine with "app_offline.htm", but not "App_Offline.htm". This is apparently fixed in a later version of the ASP.NET Core Module for IIS, but this isn't something that can exactly be installed on an Azure App Service node, and the "Rename locked files" option doesn't appear in VSTS.

Setting the 'MSDEPLOY_RENAME_LOCKED_FILES' app setting does the trick, but ideally it would be better to display a message to end users when the service is being updated.

[vincent1173]

@davidebbo, @pan-wang , to comment on the issue. The casing issue in app_offline is fixed long back.

[pan-wang]

@ColinM9991 the casing issue was fixed this Jan. When did you hit this casing issue last time?

[ColinM9991-zz]

@PanWang, it was hit both on Sunday, and yesterday (Monday).

[fmdufour]

@pan-wang I'm hitting this issue right now.

[pan-wang]

@fmdufour could you please collect server side windows event log (during the deploy) through portal and share it ?

[ColinM9991-zz]

@pan-wang

I've retried following an environment rebuild and it seems to be working now, I will put it under a bit of load once I am at home again and attempt a redeployment.

[PaitoAnderson]

Encountering this error as well today about an hour ago, it wasn't happening for over a month though.

I do have MSDEPLOY_RENAME_LOCKED_FILES enabled, and Sync Retry Attempts at 20.

[fmdufour]

@pan-wang Are you refering to eventlog.xml file? I couldn't find any relevant information on it.

[TroySchmidt]

I updated to .NET Core v1.1.4 and that seems to have fixed it. It does appear that I was having the casing issue otherwise.

[hoetz]

Hit this today as well with a aspnetcore 2.0 azure web app and I even enabled the rename locked file option. Thing is it worked at some point but now it doesn't.

[vincent1173]

@hoetz , Can you please add -retryAttempts:10 -retryInterval:6000 in additional arguments of Web Deploy options and rerun the release? Please share the debug logs for the same.

[nphmuller]

@vincentdass while this worked for me, this is seen as a workaround, right? Are you still interested in the debug logs, even with this workaround implemented? The issue didn’t occur every deployment for me, so I’m not sure if the log is still useful with the workaround. Got an email address I can mail confidential logs to?

[vincent1173]

@nphmuller , thanks for the response. Wanted to confirm from the logs, if the issue is raised by any Web jobs or DLL. Can you please share only the ASPCore version?

[hoetz]

Setting -retryAttempts:10 -retryInterval:6000 seemed to work for now

[PaitoAnderson]

Still getting this error occasionally, anything else I can try?

It really bad because it takes down our site and says "Site Under Construction" until I can click "New Build" in AppVeyor and it completes :(

I am using AppVeyor with these deploy options:

- provider: WebDeploy
  server: https://XXX.scm.azurewebsites.net:443/msdeploy.axd?site=XXX
  website: XXX
  username: $XXX
  password:
    secure: XXX
  aspnet_core: true
  app_offline: true
  sync_retry_attempts: 10
  sync_retry_interval: 6000
  remove_files: true

Error: Web Deploy cannot modify the file 'XXX.exe' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.

[pan-wang]

@PaitoAnderson you hit "Site Under Construction" is because web deploy dropped app_offline.htm file which is to signal ANCM to shutdown the backend process so that Web Deploy can overwrite the targeted assemblies. ANCM should have already sent shutdown signal to the backend. Could you please share the windows event log from the server

[vhavman]

@vincentdass Hi Vincent, we are facing the same issue which is going on in this thread, I am getting FILE_IN_USE error even after trying the following approach.

1. 

2. 

3. 

Now all these are in place for my deployment process but still getting same error. Please let me know if anything is missing or am i doing anything in wrong way.

Thanks in advance.

[jeffputz]

This sure seems like a common problem for there not to be any solution, or feedback from the VSTS team to at least say, "We're working on it."

[mmulhearn]

Also seeing this on multiple deployments.

"Web Deploy cannot modify the file 'VBCSCompiler.exe' on the destination because it is locked by an external process"

[pan-wang]

Is .net core application? Was app-offline option enabled from the deployment portal?

[mmulhearn]

No, it's a framework web app.

No, it was not.

[PaitoAnderson]

@pan-wang It's an App Service, is Event Log an option in Kudu or something?

Also, for the last 11 days I have been using this option in AppVeyor:

aspnet_core_force_restart: true

and I haven't hit a file lock since, could be a coincidence I am not sure.

[dylan-smith]

FYI, I'm getting this on a pretty regular basis (every other deploy or so), and my app is NOT core.

ASP.Net WebApi 2 (not aspnet core), deploying to azure app service. Usually the file in use is VBCSCompiler.exe

Happy to let you poke around in my account (dylanfromwinnipeg "at" gmail.com)

[vincent1173]

@dylan-smith , Is the issue happening with App Offline enabled? Can u add -retryAttempts:10 -retryInterval:6000 in additional arguments of web deploy option?

[jeffputz]

The extra arguments make no difference for me.

[dylan-smith]

@vincentdass I had app offline, and the rename file thing enabled. I added the retry args and it didn't fix anything.

[davidebbo]

@dylan-smith please see https://github.com/projectkudu/kudu/wiki/Dealing-with-locked-files-during-deployment#for-msdeploy-try-enabling-app-offline, and you the steps to determine if you're in case 1 (app_offline.htm is ignored) or in case 2 (it's taking a while).

[pan-wang]

I think the problem might be that Roslyn compiler (VBCSCompiler.exe) was running as out-of-process (This is a new compiler feature compared with the old in-process csc compiler). App_offline won't help for this scenario. App_offline only unloads app domain for asp.net application but cannot kill the child Roslyn process. This is why user still hit the file still in use issue even after enabling app_offline.
User has to fully stop the the application (IIS application pool) before deploying the new VBCSCompiler.exe.

[davidebbo]

I would still expect VBCSCompiler.exe to shut down eventually. @dylan-smith please do go through the steps I highlighted above which will give us useful data.

[Jinhuafei]

@mmulhearn @dylan-smith By default, the idle time of VBCSCompiler.exe is 10 seconds. If you find the process is idle way more than 10 seconds, please share more details about how you publish the app and go to Kudo powershell console to run the following cmd to get the Environment variable.

(get-process VBCSCompiler).StartInfo.Environment

[IvanAlekseev]

Having this configuration in asp.net core 2.0 project works fine for us now

[hkusulja]

We did not change anything, and problem started about 1-2 months ago, but it recently stopped occurring, so from my side it works. (same asp.net core 1 and 2 projects)...,

[frankfuu]

Having the same problem here -- sometimes it is bin\roslyn\csc.exe and other random dll files but mainly bin\roslyn\csc.exe

2017-11-19T16:20:25.5530009Z ==============================================================================
2017-11-19T16:20:25.5530009Z Task         : Azure App Service Deploy
2017-11-19T16:20:25.5530009Z Description  : Update Azure WebApp Services On Windows, Web App On Linux with built-in images or docker containers, ASP.NET, .NET Core, PHP, Python or Node based Web applications, Function Apps, Mobile Apps, Api applications, Web Jobs using Web Deploy / Kudu REST APIs
2017-11-19T16:20:25.5530009Z Version      : 3.3.25
2017-11-19T16:20:25.5530009Z Author       : Microsoft Corporation
2017-11-19T16:20:25.5530009Z Help         : [More Information](https://aka.ms/azurermwebdeployreadme)
2017-11-19T16:20:25.5530009Z ==============================================================================

2017-11-19T16:21:35.9113361Z Info: Updating file (myapp123\bin\roslyn\csc.exe).
2017-11-19T16:21:36.1193239Z ##[error]Failed to deploy web package to App Service.
2017-11-19T16:21:36.1213241Z ##[warning]Try to deploy app service again with Rename locked files option selected.
2017-11-19T16:21:36.1213241Z ##[error]Error Code: ERROR_FILE_IN_USE
More Information: Web Deploy cannot modify the file 'csc.exe' on the destination because it is locked by an external process.  In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.
Error count: 1.

2017-11-19T16:21:36.1233254Z ##[error]Error: C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe failed with return code: 4294967295
2017-11-19T16:21:37.1842618Z Successfully updated deployment History at https://*****.scm.azurewebsites.net/api/deployments/2451511108496095

Is there a temporary work around for this? I have to keep clicking "Redeploy" everytime I get a failed notification email

[deanilvincent]

I had the same issue. I'm using .net core 2 and in the first cd, it works fine but when I've tried to update a simple file, then all the following releases are showing the same error ... ERROR_FILE_IN_USE ...

But I fixed it by simply doing these steps

• Go to your CD definition
• Click "Tasks" (right side of the Pipeline) and choose your production name
• Choose "Deploy Azure App Service"
• Expand the "Additional Deployment Options" and check the following.

• Lastly, save the definition

I hope this may be fixed your issue :)

[vincent1173]

@frankfuu , Can you please make below changes to your task?

If you are still facing the issue, we suggest you to stop the app service, deploy and start the app service. For zero down-time deployment, you can deploy to your staging slot and swap it with production. Azure App service Manage task can be handy here.

[frankfuu]

Hey @vincentdass, I'll give it ago, but I can't consistently reproduce the issue. We already have slot swapping for our production environments so it's ok. I just want this to be implemented for our UAT/CI environment without a swapping mechanism. I'm just hoping the additional arguments -retryAttempts:10 -retryInterval:6000 will do the trick