Appservice deployment failed on 1811

[joanne03mama]

[512C:3570][2019-01-04T21:56:13]: [Websites]: TryLogAppServiceResourceStatus Ends [512C:386C][2019-01-04T21:56:13]: [Websites]: System.AggregateException: One or more errors occurred. ---> System.Exception: Deployment Failed. Refer deployment logs for more details at Microsoft.Web.Hosting.SingleInstaller.Logic.SilentDeployLogic.da.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at System.Threading.Tasks.Task.Wait() at Microsoft.Web.Hosting.SingleInstaller.WixBA.ProcessCommandLine() at Microsoft.Web.Hosting.SingleInstaller.WixBA.Run() ---> (Inner Exception #0) System.Exception: Deployment Failed. Refer deployment logs for more details at Microsoft.Web.Hosting.SingleInstaller.Logic.SilentDeployLogic.da.MoveNext()<---

[512C:4BA4][2019-01-04T21:56:13]: Shutting down, exit code: 0xffffffff [512C:4BA4][2019-01-04T21:56:13]: Variable: LocalAppDataFolder = C:\Users\AzureStackAdmin\AppData\Local\ [512C:4BA4][2019-01-04T21:56:13]: Variable: NETFRAMEWORK45 = 394802 [512C:4BA4][2019-01-04T21:56:13]: Variable: WixBundleAction = 4 [512C:4BA4][2019-01-04T21:56:13]: Variable: WixBundleElevated = 1 [512C:4BA4][2019-01-04T21:56:13]: Variable: WixBundleLog = D:\ASDKfiles\ASDK\appservice\AppServiceLog0104-163456.txt [512C:4BA4][2019-01-04T21:56:13]: Variable: WixBundleName = Microsoft Azure App Service [512C:4BA4][2019-01-04T21:56:13]: Variable: WixBundleOriginalSource = D:\ASDKfiles\ASDK\appservice\appservice.exe [512C:4BA4][2019-01-04T21:56:13]: Variable: WixBundleProviderKey = {a51eff0b-eb3b-4971-a350-ae1691850bbd} [512C:4BA4][2019-01-04T21:56:13]: Variable: WixBundleTag = [512C:4BA4][2019-01-04T21:56:13]: Variable: WixBundleVersion = 78.0.5.0 [512C:4BA4][2019-01-04T21:56:13]: Exit code: 0xffffffff, restarting: No

[mattmcspirit]

Most likely a timeout - can you share this file: D:\ASDKfiles\ASDK\appservice\AppServiceLog0104-163456.txt and a few details of your hardware?

SSDs, HDDs, how many etc?

[joanne03mama]

Deployment Log

Failed CN0-VM

Additional error information is available for this virtual machine:

General Provisioning state OS provisioning complete. Failed to provision VM extensions for VM 'CN0-VM'

Provisioning state error code ProvisioningState/failed/VMExtensionProvisioningError/osProvisioningComplete

Guest agent Ready. GuestAgent is running and accepting new configurations.

Disks osdisk Provisioning Succeeded.

Extensions OnStart Provisioning Succeeded. Finished executing command

Hardware Spec: CPU: X7550 @ 2.00GHz 2.00 GHz (16 processors) Memory: 192 GB HDD: Virtual HHD on ESXi 6.5 (1x800GB, 4x400GB)

AppServiceLog0104-163456.txt

[mattmcspirit]

Thanks - are there SSDs in your ESXi host at all?

We've seen timeouts on lower powered hardware, and while your physical host may be powerful, virtualizing the Azure Stack ASDK in an ESXi VM will naturally introduce some overhead.

In addition, we've seen timeouts where the Windows Updates have taken too long to apply to the Guests while the App Service is deploying - App Service 1.4, as per here: https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-app-service-release-notes-update-four, applies the following:

Updates to underlying operating system of all roles: 2018-10 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4462928)

The strange thing is, my images that the ASDK Configurator creates, always have the latest Cumulative Update installed when they are created, so the 2018-10 update above, shouldn't be needed.

I'll investigate with the App Service team, but in the meantime, this may help: https://rzetelnekursy.pl/azure-stack-app-service-install-problems/, the page starts in Polish I believe, but changes to English if you scroll down.

I haven't tried it myself, but if you follow steps 1-6, then rerun the script, hopefully it will continue from there.

Thanks.

[joanne03mama]

Thanks - Not SSDs in my ESXi host.

Failed to connect the CN0-VM with RDP for the method: https://rzetelnekursy.pl/azure-stack-app-service-install-problems/.

Thanks

[mattmcspirit]

Browse to Azure stack Admin portal Search for Network security group ->ControllerNsg -> Inbound security rules -> click on Inbound_Rdp_3389 -> Action -Allow

I suspect you're just running really low on resources. By the time you're installing the App Service, you're already running 15+ VMs inside this single nested ASDK host, and with no SSDs on your physical host, I suspect your host is struggling.

[joanne03mama]

Thanks VM-CN0 Web Farm error log

rror 1/5/2019 4:39:31 PM Controller heartbeat error: System.Runtime.InteropServices.COMException (0x80070013): The media is write protected. (Exception from HRESULT: 0x80070013) at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) at Microsoft.Web.Hosting.Common.NetworkHelper.GetComputerName(ComputerNameFormat format) at Microsoft.Web.Hosting.SiteManager.GetCurrentControllerName() at HostingController.ControllerOwnershipManager.IsControllerVersionUpToDate(String& controllerName) at HostingController.ControllerOwnershipManager.OnHeartBeat(Object unused) Error 1/5/2019 4:39:31 PM Web farm service encountered error: Value cannot be null. Parameter name: path1 Error 1/5/2019 4:39:31 PM System.Runtime.InteropServices.COMException (0x80090017): Provider type not defined. (Exception from HRESULT: 0x80090017) at System.Reflection.RuntimeAssembly.IsFrameworkAssembly(AssemblyName assemblyName) at System.Reflection.RuntimeAssembly.get_Flags() at System.Reflection.RuntimeAssembly.IsFrameworkAssembly() at System.Reflection.RuntimeAssembly.InternalGetSatelliteAssembly(String name, CultureInfo culture, Version version, Boolean throwOnFileNotFound, StackCrawlMark& stackMark) at System.Resources.ManifestBasedResourceGroveler.GetSatelliteAssembly(CultureInfo lookForCulture, StackCrawlMark& stackMark) at System.Resources.ManifestBasedResourceGroveler.GrovelForResourceSet(CultureInfo culture, Dictionary2 localResourceSets, Boolean tryParents, Boolean createIfNotExists, StackCrawlMark& stackMark) at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo requestedCulture, Boolean createIfNotExists, Boolean tryParents, StackCrawlMark& stackMark) at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo culture, Boolean createIfNotExists, Boolean tryParents) at System.Resources.ResourceManager.GetString(String name, CultureInfo culture) at Microsoft.Web.Hosting.ExceptionHelper.ExceptionToString(Exception exception) at Microsoft.Web.Hosting.CryptoHelper.CallCryptoMethodWithLogging[T](Func1 method, CryptoOperation operation, CryptoAlgorithm algorithm) at Microsoft.Web.Hosting.ProtectedValue.get_UnprotectedValue() at HostingController.ConfigurationHelper.VerifyADAdminCredentials(SiteManager manager) at HostingController.Extensions.MiscellaneousInitializationControllerExtension.RunWithRetries() at HostingController.Extensions.ResilientSingleRunControllerExtension.OnTimerElapsed(Object unused)

[mattmcspirit]

The thing to bear in mind here is that my script simply calls the appservice.exe, which actually handles the installation of the App Service, so I’m not sure what is causing your error above.

If you’ve managed to RDP into CN0-VM, have you replaced the status file as described here - https://rzetelnekursy.pl/azure-stack-app-service-install-problems/ ? If not, I would try that, otherwise, re-run my script, it should clean everything up and try to redeploy the App Service clean again. If it fails again, I suspect your issue is your resources but the App Service logs may not be definitive in showing that, which, I’m afraid, is out of my control.

[mattmcspirit]

Tomorrow, I’ll provide some guidance on how you can manually try to install the AppService.exe to eliminate my script causing the issue. - I’ll reply when I have those details.

[mattmcspirit]

So if you do want to manual install of the App Service, to rule out the script, here's what you can do, from an administrative PowerShell ISE:

# Login to Azure Stack
$ArmEndpoint = "https://adminmanagement.local.azurestack.external"
Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "$ArmEndpoint" -ErrorAction Stop
Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" -ErrorAction Stop
$azsLocation = (Get-AzsLocation).Name

# Clean Database
$VMpwd = Read-Host "Insert your password you used for -VMpwd when you originally ran the script"
$secureVMpwd = ConvertTo-SecureString -AsPlainText $VMpwd -Force
$SQLServerUser = "sa"
$sqlAppServerFqdn = "sqlapp.local.cloudapp.azurestack.external"
$dbCreds = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $SQLServerUser, $secureVMpwd -ErrorAction Stop
$appServiceDBCheck = Get-SqlInstance -ServerInstance $sqlAppServerFqdn -Credential $dbCreds | Get-SqlDatabase | Where-Object {$_.Name -like "*appservice*"}
foreach ($appServiceDB in $appServiceDBCheck) {
    Write-Host "$($appServiceDB.Name) database found. Cleaning up to ensure a successful rerun of the AppService deployment"
    $cleanupQuery = "ALTER DATABASE $($appServiceDB.Name) SET SINGLE_USER WITH ROLLBACK IMMEDIATE; DROP DATABASE $($appServiceDB.Name)"
    Invoke-Sqlcmd -Server $sqlAppServerFqdn -Credential $dbCreds -Query "$cleanupQuery" -Verbose 
}

$appServiceLoginCheck = Get-SqlLogin -ServerInstance $sqlAppServerFqdn -Credential $dbCreds -Verbose: $false | Where-Object {$_.Name -like "*appservice*"}
foreach ($appServiceLogin in $appServiceLoginCheck) {
    Write-Host "$($appServiceLogin.Name) login found. Cleaning up"
    Remove-SqlLogin -ServerInstance $sqlAppServerFqdn -Credential $dbCreds -LoginName $appServiceLogin.Name -Force -Verbose
}

# Clean resource group
$appServiceFailCheck = (Get-AzureRmResourceGroupDeployment -ResourceGroupName "appservice-infra" -Name "AppService.DeployCloud" -ErrorAction SilentlyContinue)
if ($appServiceFailCheck.ProvisioningState -eq 'Failed') {
    Write-Output "There is evidence of a previously failed App Service deployment in the App Service Resource Group. Starting cleanup..."
    Get-AzureRmResourceGroup -Name "appservice-infra" -Location $azsLocation -ErrorAction SilentlyContinue | Remove-AzureRmResourceGroup -Force -ErrorAction SilentlyContinue -Verbose
}

Write-Host " Log back into the portal and check the appservice-infra resource group has gone"

You're then at a point where you can manually install the App Service, as per the steps here: https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-app-service-deploy. Key information you'll need includes:

• Step 8 = \\appservicefileshare.local.cloudapp.azurestack.external\Websites, with FileShareOwner and FileShareUser as the usernames, and the password will be the same -VMpwd you used during the initial run of the script.
• Step 9 = The application ID will be in a txt file, in your -downloadPath (likely D:\ASDKfiles if you copied my steps exactly)
• Step 9 = The certificates will all be in the appservice folder, within your -downloadPath, so likely D:\ASDKfiles\ASDK\appservice and the cert password will be the same as -VMpwd Step 11 = sqlapp.local.cloudapp.azurestack.external and your user is sa, password is -VMpwd again. Step 12 = Choose a single instance for each role. By default, my script uses Standard_A2 for all instances.

You should then be able to start the deployment.

Good luck!

[joanne03mama]

Thanks. Will try to manually install app service.

[joanne03mama]

Successfully deployed app service with replaced the ESXi disk from HHD to SSD and re-configured ESXi scsi controller with scsi bus sharing enabled and Eagerthick with multi-writer enabled independent.

[mattmcspirit]

Great! Closing the issue for now - thanks for letting me know.