App Service Migration timeout on $configPathsSection = $Config.GetSection("configPaths")

JohnNicCore commented 2 months ago

Bug Report

Investigative Information

UTC Timestamp: 14/06/2024 15:00
Migration Tool App Version: 1.0.6
OS Version: 20348.2461

Repro Steps

To reproduce the problem, follow these steps:

Run the Azure App Service Migration Assistant.
Observe that the first step takes hours to complete.
Eventually, a list of web applications is displayed for selection.
Attempt to generate results, which fails.

A screenshot illustrating the issue:

Additional Details

File Size: Our applicationHost.config file is 91KB in size.

I tried using the Get-SiteReadiness.ps1 script as an alternative.

After stepping through the PowerShell scripts, I identified that the issue occurs at line 9 within the IISDiscovery.ps1 script. The script hangs indefinitely when it reaches the following line:


  $sm = New-Object Microsoft.Web.Administration.ServerManager
  $Config = $sm.GetApplicationHostConfiguration()
  $configPathsSection = $Config.GetSection("configPaths")

krolson commented 2 months ago

Hi @JohnNicCore,

Unfortunately both the App Service Migration Assistant GUI and migration PowerShell scripts rely on the same underlying API to read the IIS configuration: Microsoft.Web.Administration, which is included with IIS. In this case the issue seems to be in one of those underlying API calls to get configuration information. While the applicationHost.config file may not be large, I wonder if this could be related to the size/shape of the site content for all sites on the server. Since IIS configuration information could be at any level of the web site content, getting configuration information can mean effectively scanning through all the virtual directories and sub directories checking for the existence of web.config files, which can take more and more time the more content/sub-directories are present.

Is this possibly the case here that site content is large/contains many directories?

JohnNicCore commented 2 months ago

Hi @JohnNicCore,

Unfortunately both the App Service Migration Assistant GUI and migration PowerShell scripts rely on the same underlying API to read the IIS configuration: Microsoft.Web.Administration, which is included with IIS. In this case the issue seems to be in one of those underlying API calls to get configuration information. While the applicationHost.config file may not be large, I wonder if this could be related to the size/shape of the site content for all sites on the server. Since IIS configuration information could be at any level of the web site content, getting configuration information can mean effectively scanning through all the virtual directories and sub directories checking for the existence of web.config files, which can take more and more time the more content/sub-directories are present.

Is this possibly the case here that site content is large/contains many directories?

Hi,

Thanks for getting back to me.

There are 6 sites, and within those, there is one with 43 different virtual directories, for a total of 51 virtual directories.

It sounds like this could be our problem?

Could I modify this to make it more selective on the virtual directories I'm interested in? Some of them just hold images.

$sm = New-Object Microsoft.Web.Administration.ServerManager
$Config = $sm.GetApplicationHostConfiguration()
$configPathsSection = $Config.GetSection("configPaths")

krolson commented 2 months ago

That may be why it is hanging - we have seen cases where the call will actually come back many hours later - that's it just really slow in some types of content configurations.

There is a way to modify the script to get the configPaths one-site-at-a-time rather than all at once, but there's no guarantee it will work at all even for the site level (or it may take many hours). If you try saving below block to a file called testScript.ps1 and running it like below, this will iterate through the sites and should at least give an idea of which site(s) is hanging. The transcript is just to save the output to a file in case you lose track of the window. You may consider letting it run overnight in case it is a case of "takes a very very long time but does eventually complete".

Start-Transcript -Path .\outputResults.txt .\testScript.ps1 Stop-Transcript

If iterating per-site works where all-server does not, we can share a way to modify the IISDiscovery to use this iterative config instead. However, note that IISDiscovery will only provide a single result per site (it won't evaluate the 40+ sub-applications individually). If you suspect a particular sub-app/vdir as potentially causing the slow-down, and it is an option for your setup - you could temporarily remove the application reference from the site (in apphostconfig) to see if discovery runs when it is not included (this will make the app not-available for any traffic trying to reference it as well!).

BELOW THIS LINE AS testScript.ps1: ----------------------------------------------------------------------------

$ErrorActionPreference = "Stop"; #Make all errors terminating try {
Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : START"

    $PSVersionTable

    #LoadMWH
    $iisInstallPath = [System.Environment]::ExpandEnvironmentVariables("%windir%\system32\inetsrv\Microsoft.Web.Administration.dll");
    [System.Reflection.Assembly]::LoadFrom($iisInstallPath) | Out-Null; 
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : MWA loaded"
    try { 
        $maxMemoryInMBValue = (Get-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB).Value
        Write-Output "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : Max memory (MB): $maxMemoryInMBValue"
    } catch {
        Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : Error getting MaxMemoryPerShellMB value: $($_.Exception.ToString().Substring(0,100))"                       
    }
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : Get ServerManager start"          
    $sm = New-Object Microsoft.Web.Administration.ServerManager;
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : Get applicationHost config" 
    $Config = $sm.GetApplicationHostConfiguration();   

    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : Get site list"      
    $allSites = @();     
    $SitesSection = $Config.GetSection("system.applicationHost/sites");

    $siteCount = 0;
    $allConfigPaths = 0;
    $appHostConfigPaths = 0;

foreach($siteSection in $SitesSection.GetCollection()) {          
    $siteName = $siteSection['name'];           
    $siteCount++        
    try {
        Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : GetConfigPathSections start $siteName ($allConfigPaths) ************************************************************************************"
        $configPathsSectionIter = $Config.GetSection("configPaths",$siteName)
        Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : GetConfigPathSections for $siteName complete"
        foreach($configElement in $configPathsSectionIter.GetCollection()) {                    
            $pathValue =  $configElement['path'];
            $locationPath = $configElement['locationPath'];    
            Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : Found config at $pathValue _ $locationPath"
            if($pathvalue -ne "") {                        
                $allConfigPaths++
                if($pathValue -eq "MACHINE/WEBROOT/APPHOST") {
                    $appHostConfigPaths++
                }
            }   
        }
    } catch {
        Write-Host "********************************** $siteName ISSUE***********************************************"        
        Write-Host "Exception occurred getting config related to site $siteName : $($_.Exception)"                                                            
    }

}
Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : GetConfigPaths count complete"
Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : Number of sites: $siteCount" 
Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : Number of config paths: $allConfigPaths"
Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : Number of appHost (locationPaths): $appHostConfigPaths"

Write-Host "END"

} catch { Write-Host "Exception occurred: $($_.Exception)"
} Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") : END"

JohnNicCore commented 2 months ago

Great, thank you very helpful! I was able to narrow down the issue further with that.

It turns out the problem was related to some virtual directories pointing to a file server with over 200,000 files. After I removed these virtual directories and re-ran the migration tool, the process was completed almost instantly.

Thank you for your support.

Azure / App-Service-Migration-Assistant