Closed IvanAlekseev closed 5 years ago
@IvanAlekseev , Can you please share the full logs for the release?
@vincentdass added more info. Will it be enough?
@IvanAlekseev , Is there any web jobs enabled in App Service (including Application Insights)?
Did you enable Rename locked files
option too?
Please share your app service name without reporting it explicitly
@vincentdass - this is a dummy service name - http://errorfileinuse-msexample.azurewebsites.net and we have a problem with recommendations api.
We have app insights enabled and no other webjobs.
I will try with Rename locked files
option and tell you the results, for now we deploy to staging and then swap with prod
Same issue here: http://dummy-fileinuse.azurewebsites.net/
Happens on both the staging and production variant of the api app. I have Rename locked files
and Take App Offline
enabled. Still get the error. Retrying the deployment usually works.
Application Insights is enabled in our application. Not as an App Service extensions. No Web Jobs either.
I have tried -retryAttempts:10
but not -retryInterval:6000
. Will probably work as a workaround when I try the deployment with both.
could you please collect event log from AppService portal and share it? In general, a windows event will be logged once the file change of app_offline.htm is received and the IIS component try to gracefully shutdown the backend asp.net process.
There are at least three versions of this issue floating around, all with workarounds that don't work for everyone. It became an issue again with .NET Core v2 release time... perhaps the thing that changed has something to do with it? Having to change your build process and endure a lot of pain with every release isn't great for productivity.
Ok let's just talk here :) i have the same issue, workaround is manual turn off the slot.. i don't like that kills productivity ..
It worked for me before, and it stopped working about 2 months ago, and still have issues. My app is asp.net core v1 including the asp.net 4.6.1 , I did not change my app or deployment procedure. Issue still persist. Second build always passes. I have same issue for multiple VS TS projects / and azure web sites.
@hkusulja , Is App offline flag enabled?
Can you please try adding -retryAttempts:10 -retryInterval:6000
in Additional arguments of Web Deploy?
In this way, we increase the retry time for deployment.
I did that and it had no effect.
Having the same issue. Using a continuous deployment release definition in VSTS to AWS instance running IIS 8.5.9600.x + Windows 2012 R2 + .NET Core 1.1.2 x64. It randomly fails deploying because the file is still in use. I have tried the retryAttemps, retryInterval, put stop and start management tasks around the deploy. This is still happening regardless of any workarounds other than manually redeploy until it works.
Same issue here
Having the same issue on an Azure App Service environment with no fix on the retryAttempts and retryInterval.
It works fine with "app_offline.htm", but not "App_Offline.htm". This is apparently fixed in a later version of the ASP.NET Core Module for IIS, but this isn't something that can exactly be installed on an Azure App Service node, and the "Rename locked files" option doesn't appear in VSTS.
Setting the 'MSDEPLOY_RENAME_LOCKED_FILES' app setting does the trick, but ideally it would be better to display a message to end users when the service is being updated.
@davidebbo, @pan-wang , to comment on the issue. The casing issue in app_offline is fixed long back.
@ColinM9991 the casing issue was fixed this Jan. When did you hit this casing issue last time?
@PanWang, it was hit both on Sunday, and yesterday (Monday).
@pan-wang I'm hitting this issue right now.
@fmdufour could you please collect server side windows event log (during the deploy) through portal and share it ?
@pan-wang
I've retried following an environment rebuild and it seems to be working now, I will put it under a bit of load once I am at home again and attempt a redeployment.
Encountering this error as well today about an hour ago, it wasn't happening for over a month though.
I do have MSDEPLOY_RENAME_LOCKED_FILES enabled, and Sync Retry Attempts at 20.
@pan-wang Are you refering to eventlog.xml file? I couldn't find any relevant information on it.
I updated to .NET Core v1.1.4 and that seems to have fixed it. It does appear that I was having the casing issue otherwise.
Hit this today as well with a aspnetcore 2.0 azure web app and I even enabled the rename locked file option. Thing is it worked at some point but now it doesn't.
@hoetz , Can you please add -retryAttempts:10 -retryInterval:6000
in additional arguments of Web Deploy options and rerun the release?
Please share the debug logs for the same.
@vincentdass while this worked for me, this is seen as a workaround, right? Are you still interested in the debug logs, even with this workaround implemented? The issue didn’t occur every deployment for me, so I’m not sure if the log is still useful with the workaround. Got an email address I can mail confidential logs to?
@nphmuller , thanks for the response. Wanted to confirm from the logs, if the issue is raised by any Web jobs or DLL. Can you please share only the ASPCore version?
Setting -retryAttempts:10 -retryInterval:6000 seemed to work for now
Still getting this error occasionally, anything else I can try?
It really bad because it takes down our site and says "Site Under Construction" until I can click "New Build" in AppVeyor and it completes :(
I am using AppVeyor with these deploy options:
- provider: WebDeploy
server: https://XXX.scm.azurewebsites.net:443/msdeploy.axd?site=XXX
website: XXX
username: $XXX
password:
secure: XXX
aspnet_core: true
app_offline: true
sync_retry_attempts: 10
sync_retry_interval: 6000
remove_files: true
Error:
Web Deploy cannot modify the file 'XXX.exe' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.
@PaitoAnderson you hit "Site Under Construction" is because web deploy dropped app_offline.htm file which is to signal ANCM to shutdown the backend process so that Web Deploy can overwrite the targeted assemblies. ANCM should have already sent shutdown signal to the backend. Could you please share the windows event log from the server
@vincentdass Hi Vincent, we are facing the same issue which is going on in this thread, I am getting FILE_IN_USE error even after trying the following approach.
Now all these are in place for my deployment process but still getting same error. Please let me know if anything is missing or am i doing anything in wrong way.
Thanks in advance.
This sure seems like a common problem for there not to be any solution, or feedback from the VSTS team to at least say, "We're working on it."
Also seeing this on multiple deployments.
"Web Deploy cannot modify the file 'VBCSCompiler.exe' on the destination because it is locked by an external process"
Is .net core application? Was app-offline option enabled from the deployment portal?
No, it's a framework web app.
No, it was not.
@pan-wang It's an App Service, is Event Log an option in Kudu or something?
Also, for the last 11 days I have been using this option in AppVeyor:
aspnet_core_force_restart: true
and I haven't hit a file lock since, could be a coincidence I am not sure.
FYI, I'm getting this on a pretty regular basis (every other deploy or so), and my app is NOT core.
ASP.Net WebApi 2 (not aspnet core), deploying to azure app service. Usually the file in use is VBCSCompiler.exe
Happy to let you poke around in my account (dylanfromwinnipeg "at" gmail.com)
@dylan-smith , Is the issue happening with App Offline enabled?
Can u add -retryAttempts:10 -retryInterval:6000
in additional arguments of web deploy option?
The extra arguments make no difference for me.
@vincentdass I had app offline, and the rename file thing enabled. I added the retry args and it didn't fix anything.
@dylan-smith please see https://github.com/projectkudu/kudu/wiki/Dealing-with-locked-files-during-deployment#for-msdeploy-try-enabling-app-offline, and you the steps to determine if you're in case 1 (app_offline.htm is ignored) or in case 2 (it's taking a while).
I think the problem might be that Roslyn compiler (VBCSCompiler.exe) was running as out-of-process (This is a new compiler feature compared with the old in-process csc compiler).
App_offline won't help for this scenario. App_offline only unloads app domain for asp.net application but cannot kill the child Roslyn process. This is why user still hit the file still in use issue even after enabling app_offline.
User has to fully stop the the application (IIS application pool) before deploying the new VBCSCompiler.exe.
I would still expect VBCSCompiler.exe to shut down eventually. @dylan-smith please do go through the steps I highlighted above which will give us useful data.
@mmulhearn @dylan-smith By default, the idle time of VBCSCompiler.exe is 10 seconds. If you find the process is idle way more than 10 seconds, please share more details about how you publish the app and go to Kudo powershell console to run the following cmd to get the Environment variable.
(get-process VBCSCompiler).StartInfo.Environment
Having this configuration in asp.net core 2.0 project works fine for us now
We did not change anything, and problem started about 1-2 months ago, but it recently stopped occurring, so from my side it works. (same asp.net core 1 and 2 projects)...,
Having the same problem here -- sometimes it is bin\roslyn\csc.exe and other random dll files but mainly bin\roslyn\csc.exe
2017-11-19T16:20:25.5530009Z ==============================================================================
2017-11-19T16:20:25.5530009Z Task : Azure App Service Deploy
2017-11-19T16:20:25.5530009Z Description : Update Azure WebApp Services On Windows, Web App On Linux with built-in images or docker containers, ASP.NET, .NET Core, PHP, Python or Node based Web applications, Function Apps, Mobile Apps, Api applications, Web Jobs using Web Deploy / Kudu REST APIs
2017-11-19T16:20:25.5530009Z Version : 3.3.25
2017-11-19T16:20:25.5530009Z Author : Microsoft Corporation
2017-11-19T16:20:25.5530009Z Help : [More Information](https://aka.ms/azurermwebdeployreadme)
2017-11-19T16:20:25.5530009Z ==============================================================================
2017-11-19T16:21:35.9113361Z Info: Updating file (myapp123\bin\roslyn\csc.exe).
2017-11-19T16:21:36.1193239Z ##[error]Failed to deploy web package to App Service.
2017-11-19T16:21:36.1213241Z ##[warning]Try to deploy app service again with Rename locked files option selected.
2017-11-19T16:21:36.1213241Z ##[error]Error Code: ERROR_FILE_IN_USE
More Information: Web Deploy cannot modify the file 'csc.exe' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.
Error count: 1.
2017-11-19T16:21:36.1233254Z ##[error]Error: C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe failed with return code: 4294967295
2017-11-19T16:21:37.1842618Z Successfully updated deployment History at https://*****.scm.azurewebsites.net/api/deployments/2451511108496095
Is there a temporary work around for this? I have to keep clicking "Redeploy" everytime I get a failed notification email
I had the same issue. I'm using .net core 2 and in the first cd, it works fine but when I've tried to update a simple file, then all the following releases are showing the same error
... ERROR_FILE_IN_USE ...
But I fixed it by simply doing these steps
I hope this may be fixed your issue :)
@frankfuu , Can you please make below changes to your task?
If you are still facing the issue, we suggest you to stop the app service, deploy and start the app service.
For zero down-time deployment, you can deploy to your staging slot and swap it with production.
Azure App service Manage
task can be handy here.
Hey @vincentdass, I'll give it ago, but I can't consistently reproduce the issue. We already have slot swapping for our production environments so it's ok. I just want this to be implemented for our UAT/CI environment without a swapping mechanism. I'm just hoping the additional arguments -retryAttempts:10 -retryInterval:6000
will do the trick
I am trying to deploy an Asp.Net Core 2.0 web app to Azure App Service and receive this error
I've tried some workarounds from similar issue without any luck https://github.com/Microsoft/vsts-tasks/issues/1607
Any ideas on what should I do to fix this?