"Access to the path 'Microsoft.CodeAnalysis.CSharp.dll' is denied" error when deleting bin\roslyn directory in a release pipeline after stopping IIS application pool

[stevenvolckaert]

Hello!

We're using Microsoft.CodeDom.Providers.DotNetCompilerPlatform v3.6.0 in a number of our ASP.NET Web Forms and ASP.NET MVC 5 applications, and it works well.

We deploy these applications with Azure DevOps Pipelines, and just before it deploys, the pipeline stops the IIS application pool, deletes all items in the IIS website's physical path, and finally deploys the new version. This ensures nothing of the old version stays behind in the physical path; we call this a clean deploy.

The clean is done with a PowerShell script; you can see its implementation on the bottom of this post.

We notice that sometimes, the clean step fails and some files stay behind; the Remove-Item cmdlet throws an error Access to the path 'Microsoft.CodeAnalysis.CSharp.dll' is denied:

2021-11-30T21:57:17.8887276Z ##[section]Starting: Clean D:\skarabee.net\sk7
2021-11-30T21:57:17.9004217Z ==============================================================================
2021-11-30T21:57:17.9004643Z Task         : PowerShell
2021-11-30T21:57:17.9004975Z Description  : Run a PowerShell script on Linux, macOS, or Windows
2021-11-30T21:57:17.9005318Z Version      : 2.194.0
2021-11-30T21:57:17.9005606Z Author       : Microsoft Corporation
2021-11-30T21:57:17.9006013Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
2021-11-30T21:57:17.9006467Z ==============================================================================
2021-11-30T21:57:18.9345050Z Generating script.
2021-11-30T21:57:18.9386203Z Formatted command: . 'C:\azagent\A3\_work\4\s\Remove-ItemsInPath.ps1' -Path "D:\skarabee.net\sk7" -Exclude "app_offline.htm",  "[Aa]pp[Ss]ettings.[Cc]onfig",  "[Cc]onnection[Ss]trings.[Cc]onfig",  "[Dd]ata[Cc]onfiguration*.[Cc]onfig",  "[Ee]ngine[Cc]onfiguration*.[Cc]onfig",  "[Ww]eb*.[Cc]onfig"
2021-11-30T21:57:18.9861253Z ========================== Starting Command Output ===========================
2021-11-30T21:57:19.0075343Z ##[command]"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\azagent\A3\_work\_temp\2f97630d-a052-4be6-8a86-7e1ce54d19a0.ps1'"
2021-11-30T21:57:19.3022110Z Delete files in path 'D:\skarabee.net\sk7' recursively (-Exclude 'app_offline.htm, [Aa]pp[Ss]ettings.[Cc]onfig, [Cc]onnection[Ss]trings.[Cc]onfig, [Dd]ata[Cc]onfiguration*.[Cc]onfig, [Ee]ngine[Cc]onfiguration*.[Cc]onfig, [Ww]eb*.[Cc]onfig')
2021-11-30T21:57:20.4477603Z Remove-Item : Cannot remove item D:\skarabee.net\sk7\Bin\roslyn\Microsoft.CodeAnalysis.CSharp.dll: Access to the path 
2021-11-30T21:57:20.4478380Z 'Microsoft.CodeAnalysis.CSharp.dll' is denied.
2021-11-30T21:57:20.4478661Z At C:\azagent\A3\_work\4\s\Remove-ItemsInPath.ps1:45 char:7
2021-11-30T21:57:20.4479024Z +     | Remove-Item -Recurse;
2021-11-30T21:57:20.4479462Z +       ~~~~~~~~~~~~~~~~~~~~
2021-11-30T21:57:20.4480394Z     + CategoryInfo          : PermissionDenied: (Microsoft.CodeAnalysis.CSharp.dll:FileInfo) [Remove-Item], Unauthoriz 
2021-11-30T21:57:20.4480757Z    edAccessException
2021-11-30T21:57:20.4481862Z     + FullyQualifiedErrorId : RemoveFileSystemItemUnAuthorizedAccess,Microsoft.PowerShell.Commands.RemoveItemCommand
2021-11-30T21:57:20.6010246Z ##[error]PowerShell exited with code '1'.
2021-11-30T21:57:20.6460341Z ##[section]Finishing: Clean D:\skarabee.net\sk7

What could be the cause of this? As I mentioned, the pipeline stops the application pool before the clean script runs, so I guess there must be another process running on the server that locked Microsoft.CodeAnalysis.CSharp.dll (and possibly other DLLs, too).

I guess my issue is related to issue #10, in which @Jinhuafei mentions that VBCSCompiler.exe is using the files and hence has locked the DLLs.

I've been searching the server with the Get-Process cmdlet for a process that looks like VBCSCompiler.exe (see screenshot below), but no luck.

I'd like to add a PowerShell script to the pipeline, that checks whether VBCSCompiler.exe or csc.exe is running, and kill their processes before cleaning the directory.

The goal ultimately, is that the clean step doesn't fail on these locked files any more.

Any ideas how I can achieve this?

Many thanks in advance for your help!

Implementation of the Remove-ItemsInPath.ps1 script:

#Requires -Version 3

<#
.SYNOPSIS
    Removes all files and directories from a specified path, recursively.
.DESCRIPTION
    This cmdlet removes all files and directories from the specified path, recursively. Files and directories which are
    a child of the specified path and which match one of the patterns provided with the -Exclude parameter are ignored;
    these files will not be removed.
.NOTES
    File Name       : Remove-ItemsInPath.ps1
    Author          : Steven Volckaert (steven.volckaert@gmail.com)
    Prerequisites   : PowerShell version 3
.EXAMPLE
    .\Remove-ItemsInPath.ps1 -Path "C:\temp" -Exclude "logs", "test"
#>

Param (
    [string] $Path = $(throw "Argument -Path is required"),
    [string[]] $Exclude = @("")
)

# IMPORTS #############################################################################################################

# FUNCTIONS ###########################################################################################################

# MAIN ################################################################################################################

$ofs = ", ";
$Stopwatch = New-Object -TypeName System.Diagnostics.Stopwatch;

if ((Test-Path "$Path") -eq $false) {
    Write-Error "Abort: Path '$Path' does not exist .";
    exit 1;
}

$LogMessageBase = "Delete files in path '$Path' recursively$(if ($Exclude) { " (-Exclude '$Exclude')" })";
Write-Output $LogMessageBase;
$Stopwatch.Start();

Get-ChildItem -Path $Path -Exclude $Exclude `
    | Sort-Object -Property FullName -Descending `
    | Remove-Item -Recurse;

$Stopwatch.Stop();
Write-Output "$LogMessageBase - OK ($($Stopwatch.ElapsedMilliseconds) ms)";

[stevenvolckaert]

Hello, is anyone listening? Thank you!

[CZEMacLeod]

@stevenvolckaert We use a very similar approach - although we don't use a script, just the flag in the deploy step to remove any files not matched in the zip file. RemoveAdditionalFilesFlag The other thing is we stop the site first, then the pool, then deploy. I am not aware of ever having an issue with this setup. I don't believe csc.exe or vbc.exe would be running if the application pool is stopped, but technically the VBCSCompiler.exe process could hang around. I think you should try logging with Get-Process | Where-Object ProcessName -eq 'VBCSCompiler' | Write-Output in your PowerShell script to see if you ever get a result. You could simplify this to Get-Process 'VBCSCompiler' | Write-Output and kill with Get-Process 'VBCSCompiler' | Stop-Process

[stevenvolckaert]

Hi @CZEMacLeod! Thanks for your suggestions, I'll try them out.

[StephenMolloy]

Unfortunately, we don't have a lot of control over the VBCSCompiler process. In an msbuild environment, we have a target to try and kill it before trying to re-copy the roslyn files to the output directory. But at runtime or in a release pipeline that approach doesn't work. You could always experiment with changing the TTL of the roslyn compiler server. There are instructions on the main page of this repo. (Look for "VBCSCOMPILER_TTL" or "CompilerServerTimeToLive"). Hopefully following a careful shutdown of application pools and a cleanup of any leftover VBCSCompiler processes along the lines of @CZEMacLeod's suggestions can unblock you here. Let us know if you're still having issues.

[stevenvolckaert]

Thank you all for your suggestions.

We haven't encountered this issue any more in the last three months. I'm hereby closing this issue.