IIS Deploy on premises fails with ERROR_FILE_IN_USE

[mpoiron]

Required Information

Question, Bug, or Feature?
Type: Bug

Enter Task Name: IISWebAppDeploymentOnMachineGroup

Environment

• Server - Azure Pipelines
• Agent - Hosted

Issue Description

I'm using the IISWebAppDeploymentOnMachineGroup@0 task to deploy a .NET Core 3.1 app to IIS on premises. The task fails sometimes with a ERROR_FILE_IN_USE code. I have set the TakeAppOfflineFlag to true but nothing changed.

Task logs

tasklogs.zip

Troubleshooting

Error logs

Info: Updating file (AGIL\Crpdl.Agil.Web.dll).
##[debug]Exit code 4294967295 received from tool 'C:\azagent\A1\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.156.13\node_modules\webdeployment-common-v2\MSDeploy3.6\msdeploy.exe'
##[debug]Exit code 4294967295 received from tool 'C:\azagent\A1\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.156.13\node_modules\webdeployment-common-v2\MSDeploy3.6\msdeploy.exe'
##[debug]STDIO streams have closed for tool 'C:\azagent\A1\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.156.13\node_modules\webdeployment-common-v2\MSDeploy3.6\msdeploy.exe'
##[debug]STDIO streams have closed for tool 'C:\azagent\A1\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.156.13\node_modules\webdeployment-common-v2\MSDeploy3.6\msdeploy.exe'
##[error]Failed to deploy web package to IIS website.
##[debug]Processed: ##vso[task.issue type=error;]Failed to deploy web package to IIS website.
##[debug]{}
##[debug]System.DefaultWorkingDirectory=C:\azagent\A1\_work\1\s
##[warning]Can\'t find loc string for key: Trytodeploywebappagainwithrenamefileoptionselected
##[debug]Processed: ##vso[task.issue type=warning;]Can\'t find loc string for key: Trytodeploywebappagainwithrenamefileoptionselected
##[warning]Trytodeploywebappagainwithrenamefileoptionselected
##[debug]Processed: ##vso[task.issue type=warning;]Trytodeploywebappagainwithrenamefileoptionselected
##[error]Error Code: ERROR_FILE_IN_USE
More Information: Web Deploy cannot modify the file 'Crpdl.Agil.Web.dll' on the destination because it is locked by an external process.  In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.
Error count: 1.

[arjgupta]

@mpoiron can follow try to follow the advice in this discussion here: https://github.com/aspnet/IISIntegration/issues/238 especially this comment here about using setting an app offline flag https://github.com/aspnet/IISIntegration/issues/238#issuecomment-406006574 .

[thurfir]

Hi, if this might help, even with the AppOffline flag enabled we are getting this ERROR_FILE_IN_USE exception on-premise from time to time. It started since moving to .net core 2.2 with the AspNetCoreHostingModel flag enabled. This flag default to true since 3.0 .

From the data I gathered from diverse sources this problem has been fixed in Azure by renaming locked dll(s) , hence the ticket we opened here for on-premise support in webDeploy : https://github.com/microsoft/azure-pipelines-tasks/issues/13223

[arjgupta]

@mpoiron are you still facing the issue ? can you try setting the MSDEPLOY_RENAME_LOCKED_FILES app setting and setting it's value to 1.

https://github.com/projectkudu/kudu/wiki/Dealing-with-locked-files-during-deployment#for-msdeploy-there-is-another-option-to-rename-locked-files

[thurfir]

@mpoiron are you still facing the issue ? can you try setting the MSDEPLOY_RENAME_LOCKED_FILES app setting and setting it's value to 1.

https://github.com/projectkudu/kudu/wiki/Dealing-with-locked-files-during-deployment#for-msdeploy-there-is-another-option-to-rename-locked-files

Hi, afaik MSDEPLOY_RENAME_LOCKED_FILES is for Azure, right? This is on-premise.

[mpoiron]

Hi,

I am still facing the issue, and the MSDEPLOY_RENAME_LOCKED_FILES setting indeed seems to be only for Azure and I am on-premises. Is there any news on fixing the TakeAppOffline flag or allowing "rename deployment" on premises ?

[mhgames]

Same issue here. Is there at least some workaround? I'm guessing a copy task with app_offline.htm and then delete it after publishing...

[mpoiron]

Can we have an update on this issue ? I am still having this issue and having to kill the app pool each time is quite annoying.

[Yush0]

We are facing the same problem since upgrading to 3.1 from 2.1. (on premise, InApp hosting model)

It only occurs when some longer running requests are still going on during deployment. Its quite a regression not being able to reliably hot deploy the application.

MSDEPLOY_RENAME_LOCKED_FILES is really needed for on premise too.

[asranja]

Please try using IIS Web App Manage task to stop the application pool before deploying and then start the app pool again.

- task: IISWebAppManagementOnMachineGroup@0
  inputs:
    IISDeploymentType: 'IISApplicationPool'
    ActionIISApplicationPool: 'StopAppPool'
    StartStopRecycleAppPoolName: 'Testpool'

- task: IISWebAppDeploymentOnMachineGroup@0
  inputs:
    WebSiteName: 'TestSite'
    Package: '$(System.DefaultWorkingDirectory)\**\*.zip'

- task: IISWebAppManagementOnMachineGroup@0
  inputs:
    IISDeploymentType: 'IISApplicationPool'
    ActionIISApplicationPool: 'StartAppPool'
    StartStopRecycleAppPoolName: 'Testpool'

[KuroroSam]

Please try using IIS Web App Manage task to stop the application pool before deploying and then start the app pool again.

- task: IISWebAppManagementOnMachineGroup@0
  inputs:
    IISDeploymentType: 'IISApplicationPool'
    ActionIISApplicationPool: 'StopAppPool'
    StartStopRecycleAppPoolName: 'Testpool'

- task: IISWebAppDeploymentOnMachineGroup@0
  inputs:
    WebSiteName: 'TestSite'
    Package: '$(System.DefaultWorkingDirectory)\**\*.zip'

- task: IISWebAppManagementOnMachineGroup@0
  inputs:
    IISDeploymentType: 'IISApplicationPool'
    ActionIISApplicationPool: 'StartAppPool'
    StartStopRecycleAppPoolName: 'Testpool'

We tried to this approach, it works on some deployments but not for all. We have more than 500 sites manage by Azure DevOps On-Permises. Our deployment may be locked by HTTP keep-alive connection, we have assumed StopAppPool should terminate all of IIS connections in that AppPool.

[20shivangi]

As mentioned in doc : If you have enabled App Offline and you're still seeing issues with files in use, there are two main possibilities:

1. The first is that the runtime is completely failing to detect App_Offline.htm
2. The second is that the runtime detects it, but it takes a while to kick in, and msdeploy is not waiting long enough for file locks to get releases

Please go through this doc once with the steps mentioned and let us know if this unblocks you

[20shivangi]

@mpoiron Any update here

[mpoiron]

I seem to have the 2nd problem, I'll try the retry arguments and check back if I still have any issue.

[aksm-ms]

@mpoiron Are you still facing any issues?

[Yush0]

We are still getting some rare FILE_IN_USE errors

I was able to reproduce this behaviour by running a request that contains only await Task.Delay(40000); during deployment.

But normally we get the FILE_IN_USE in another dll that handles Entity Framework matters.

[20shivangi]

Was the doc : https://github.com/projectkudu/kudu/wiki/Dealing-with-locked-files-during-deployment#for-msdeploy-try-enabling-app-offline not helpful to you ?

[Yush0]

@20shivangi No the docs don't provide a real solution for us:

"The downside of course is that your site is unavailable during the deployment. This can be avoided by relying on a Staging slot instead of deploying straight to Production."

We already have Staging Sites running but at some point we need to deploy to Production. In most cases the deployment works seamlessly without app_offline and we have no down time at all. In rare cases it fails, which we never had before 2.1. I remember having read that there were some shadow copy mechanisms in place before. MSDEPLOY_RENAME_LOCKED_FILES seems to do something similar but is only available for Azure. For us its important to be able to hot deploy an application also on premise without taking it offline. I understand that cloud is now first class citizen but this is a critical feature for on premise too.

[rafastellato]

Same problem here... I am using .NET 5 and deploying on premises.

The problem by check the option "Take app offline" is that if the deploy fail, the api goes down :(

[guinatal]

Same problem here. .NET 5 web api running on premise. It looks like this is not something new by researching it.... still didn't find a solution though.

[ssinfod]

We also have the same problem. The "Microsoft.Data.SqlClient.SNI.x64.dll" is locked.

This is giving me a deployment error with Azure Devops.

For now, the only workaround I have is to stop and start the Application Pool of the main website. I think the problem should be fixed in the framework. Please release the unmanaged resource.

[Bazul99]

We're running into this problem with msvcr120.dll and that's even stopping the Application Pool. Maybe a shared configuration and multiple websites with 2 IIS servers makes this even harder problem to solve?

[dougclutter]

We've been having this problem since .NET Core 2.2. Here's a PS script that helps us work around it:

param ( [string]$appPoolOperation = $(Throw "Valid values are RecycleAppPool, StartAppPool, StopAppPool, UnloadAppDomain."), [string[]]$servers = $(Throw "Please provide a server URL."), [string]$iisSite = $(Throw "Please provide the IIS site name."), [string]$webApp = $(Throw "Please provide IIS application name or . for root directory.") )

if ($webApp -eq ".") {

Website is in root directory

$webApp = $iisSite

} else {

Website is in web application subfolder

$webApp = Join-Path $iisSite $webApp

}

$msdeploy = Join-Path -Path (Get-ChildItem 'HKLM:\SOFTWARE\Microsoft\IIS Extensions\MSDeploy').GetValue('InstallPath') -ChildPath msdeploy.exe foreach($server in $servers) { Write-Host "##[command] Performing $appPoolOperation on $webApp on $server" $startTime = Get-Date

$dest = "recycleApp='$webApp',recycleMode=$appPoolOperation,computerName=https://$server`:8172/msdeploy.axd?site=$iisSite,username=$env:DEPLOY_USER,password=$env:DEPLOY_PASSWORD,authtype=Basic,includeAcls=False"
& $msdeploy `
    -source:recycleApp `
    -dest:$dest `
    -verb:sync `
    -allowUntrusted

$executionTime = (Get-Date).Subtract($startTime)
Write-Host "************************************ Execution End: $($executionTime.Minutes) min $($executionTime.Seconds) sec *************************************`n"

}

[freemstr]

Maybe worth revisiting with https://github.com/Microsoft/azure-pipelines-tasks/issues/9596

[DannyBoyIT]

Got this error today while deploying a hotfix that took our whole webapp down.

Is there any workarround for this in Azure Devops classic deployment? I guess one workarround should be to deploy each time to a new folder and then changing the app directory in the IIS but I don't know if I really like that.

[v-jkarri]

@DannyBoyIT , Can you please raise a new issue with detailed logs.

[sg53719]

We are also getting the ERROR_FILE_IN_USE in one of our .NET 5 DLLs that has Entity Framework Core. We have tried inserting Stop App Pool, then 60 second delay, then deploy then start without luck.

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[FreeFrags]

We have the same issue the deploys fail randomly with the File in use error (the same as attached to the issue multiple times now).

Hopefully the bot will remove the stale tag and we find a solution for this soon.

[Bazul99]

In our case I'm fairly certain that the issue stems from the fact that we're deploying to a set of IIS servers running in Shared Configuration. The deployment agent is running only on one server so a restart or reset before deployment performed by the pipeline via the agent only happens on 1/2 of the Shared Configuration so it doesn't always do the trick if the file is locked by the other server.

[FreeFrags]

We have the same issue the deploys fail randomly with the File in use error (the same as attached to the issue multiple times now).

Hopefully the bot will remove the stale tag and we find a solution for this soon.

We solved our issue by stopping the application pool then deploying and restarting the application pool

[VictorioBerra]

I am encountering this all of the sudden. Restarting the pools and retrying the deployment usually fixes it but thats a poor fix.

Please advise.

[guidupuy]

Have been facing this for months now, which is incredibly frustrating. Can we please, PLEASE have an equivalent of MSDEPLOY_RENAME_LOCKED_FILES be included in the msdeploy flags?

[VictorioBerra]

@guidupuy what I ended up doing in Azure Pipelines is going an IIS manage and recycling the pools before deployment. Not great but could be worse.

[ntndash]

Have anyone tried this solution ?

https://dev.to/lukaszreszke/solving-the-azure-devops-release-error-code-errorfileinuse-25jn

[guidupuy]

I ended up using shadow copying, which is a great way to avoid this problem and is available in .NET 6, albeit as an experimental feature.

[ranulp-fs]

I was able to get around this somewhat by using the additional argument -useChecksum in the IIS Web App Deploy task. This makes it possible to get the pipeline to succeed in most cases (i.e. when the culprit dll is not actually modified). As a fallback for situations in which it is actually modified, I added the arguments -retryAttempts:10 -retryInterval:6000 alongside -useChecksum. This works for me because after app_offline.htm is created, it takes close to 1 minute for the culprit dll to become delete-able.

[keeehlan]

My team is plagued by this issue. IIS, on-premises, App Offline file and configuration set appropriately. Frequently have to kill the app pool to deploy.

Can't believe this is still an issue with no reliable fix.

Edit: IIS on-premises, MS Web Deploy

[VictorioBerra]

@keeehlan Did you try:

create an "IIS manage" task and recycle the pool before deployment?

[keeehlan]

@VictorioBerra Isn't the whole point of the app_offline.htm file that it stops the application and removes the need for this?

Also, as I said in my comment, we are not using Azure, we are deploying to IIS using Web Deploy.

[VictorioBerra]

@keeehlan Sure. But this issue is almost 3 years old with no official help from Microsoft, you say you are doing everything right which is great but there is obviously bugs and you need a workaround. This has been working for me and other users in this thread and it takes 2 seconds to implement.

[VictorioBerra]

@keeehlan

Also, as I said in my comment, we are not using Azure, we are deploying to IIS using Web Deploy.

So you are in the wrong issue then? This issue is for people using Azure Pipelines Tasks to deploy their apps. Either up in Azure OR on-premise on IIS. Like many of us here are doing and are discussing.

[recursivegeek]

Keeping this alive as we are seeing the issue with .NET 6 web apps (a newly developed application that is about to go live). DLLs are locked when deploying to an on-premise IIS server via task IISWebAppDeploymentOnMachineGroup@0. The Azure DevOps Pipeline logs the following error while trying to update a DLL:

[error]Failed to deploy web package to IIS website.

[error]Error Code: ERROR_FILE_IN_USE

[error]Error: The process 'C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe' failed with exit code 4294967295

[bunyadahmadli]

Hello, I have same issue. @RecursiveGeek could you fix this?

IIS on-premises, MS Web Deploy

[mclausing]

I'm still experiencing this in a few different projects. Any update?

[godzkaya]

Any updates here? Still facing this issue.

[fafafooey916]

I still have this issue on various projects. Randomly, files are held by some unknown process. It's getting aggravating

[godzkaya]

We have the same issue the deploys fail randomly with the File in use error (the same as attached to the issue multiple times now). Hopefully the bot will remove the stale tag and we find a solution for this soon.

We solved our issue by stopping the application pool then deploying and restarting the application pool

So stopping the pool and restarting can work but the website will be down till deployment finishes. Which nobody would satisfy with this solution. am I right? Yet, in a technical way if a file in use by another process you cannot override the file until it stops.

[VictorioBerra]

Or you could just recycle the pool to unlock the file and perform the deployment.

On Thu, Oct 19, 2023, 3:43 AM godzkaya @.***> wrote:

We have the same issue the deploys fail randomly with the File in use error (the same as attached to the issue multiple times now). Hopefully the bot will remove the stale tag and we find a solution for this soon.

We solved our issue by stopping the application pool then deploying and restarting the application pool

So stopping the pool and restarting can work but the website will be down till deployment finishes. Which nobody would satisfy with this solution. am I right? Yet, in a technical way if a file in use by another process you cannot override the file until it stops.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/azure-pipelines-tasks/issues/13232#issuecomment-1770340951, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWMN22GPXBQGL2GR3HQMLDYADR2DAVCNFSM4OPY2W2KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZXGAZTIMBZGUYQ . You are receiving this because you were mentioned.Message ID: @.***>

[fafafooey916]

Or you could just recycle the pool to unlock the file and perform the deployment. … On Thu, Oct 19, 2023, 3:43 AM godzkaya @.> wrote: We have the same issue the deploys fail randomly with the File in use error (the same as attached to the issue multiple times now). Hopefully the bot will remove the stale tag and we find a solution for this soon. We solved our issue by stopping the application pool then deploying and restarting the application pool So stopping the pool and restarting can work but the website will be down till deployment finishes. Which nobody would satisfy with this solution. am I right? Yet, in a technical way if a file in use by another process you cannot override the file until it stops. — Reply to this email directly, view it on GitHub <#13232 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWMN22GPXBQGL2GR3HQMLDYADR2DAVCNFSM4OPY2W2KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZXGAZTIMBZGUYQ . You are receiving this because you were mentioned.Message ID: @.>

Many of us are stopping the app pool, but, many times, that does not solve the issue. Are you saying a recycle is better to use or should be used in conjunction with a stoppage?

[Maryam1986]

we have Azure Devops server 2022.1 and after updating our .net version to 8 we have also this problems with every 2 release : Failed to deploy web package to IIS website. Error Code: ERROR_FILE_IN_USE Additional Information: Web Deploy cannot modify the Localization.resources.dll file in the target because it is locked by an external process. For the publishing process to complete successfully, you may need to restart your application to release the lock or use the AppOffline rule handler for .NET applications on your next publishing attempt.