mattmcspirit / azurestack

Azure Stack Resources
80 stars 41 forks source link

Appservice deployment failed #60

Closed gijs007 closed 5 years ago

gijs007 commented 5 years ago

I've ran the post deployment script on a fresh ASDK with the following arguments:

C:\ConfigASDK> .\ConfigASDK.ps1 -azureDirectoryTenantName "xxxxx.onmicrosoft.com" -authenticationType AzureAD `
-downloadPath "D:\ASDKpostdeploytemp" -ISOPath "D:\ASDKdependencies_offlinedownload\ConfigASDKfiles\Windows_Server_20
16_Datacenter_EVAL_en-us_14393_refresh.ISO" -azureStackAdminPwd "xxxxx" `
-VMpwd "xxxx" -azureAdUsername "xxxx" -azureAdPwd "xxxx" `
-registerASDK -useAzureCredsForRegistration -azureRegSubId "xxxx" -skipMySQL -skipCustomizeHost

The post deployment step for DeployAppService failed. After checking the AppServiceLog it, it seems that a virtual machine is being deployed. But the deployment of this virtual machine failed.

From the: AppServiceLog1116-011218.txt:

"extensions": [
    {
      "name": "OnStart",
      "type": "Microsoft.Compute.CustomScriptExtension",
      "typeHandlerVersion": "1.9.1",
      "substatuses": [
        {
          "code": "ComponentStatus/StdOut/Succeeded",
          "level": "Info",
          "displayStatus": "Provisioning Succeeded",
          "message": "tabase to be ready...\\n2018.11.16 02:56:56 ---- Waiting for Hosting database to be ready...\\n2018.11.16 02:58:06 ---- Waiting for Hosting database to be ready...\\n2018.11.16 02:59:17 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:00:27 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:01:37 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:02:47 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:03:57 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:05:07 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:06:18 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:07:28 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:08:38 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:09:48 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:10:59 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:12:09 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:13:19 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:14:29 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:15:39 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:16:49 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:18:00 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:19:10 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:20:21 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:21:32 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:22:42 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:23:53 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:25:03 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:26:29 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:27:40 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:28:50 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:30:00 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:31:10 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:32:21 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:33:31 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:34:41 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:35:51 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:37:02 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:38:12 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:39:22 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:40:33 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:41:43 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:42:54 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:44:04 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:45:14 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:46:24 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:47:35 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:48:45 ---- Waiting for Hosting database to be ready...\\n<WARNING> Exception: Could not find stored procedure 'runtime.healthCheck_v88'.\\n2018.11.16 03:49:50 ---- Waiting for Hosting database to be ready...\\n<WARNING> Exception: Could not find stored procedure 'runtime.healthCheck_v88'.\\n2018.11.16 03:50:50 ---- Waiting for Hosting database to be ready...\\n<WARNING> Exception: Could not find stored procedure 'runtime.healthCheck_v88'.\\n2018.11.16 03:51:50 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:52:52 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:53:52 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:54:52 ---- Waiting for Hosting database to be ready...\\n2018.11.16 03:55:52 ---- Hosting Database is ready!\\n2018.11.16 03:56:21 ---- Start Role completed\\n2018.11.16 03:56:30 ---- App Service Role OnStart() has been configured.\\n2018.11.16 03:56:31 ---- App Service role installation has completed successfully.\\n\\n"

        },

        {
          "code": "ComponentStatus/StdErr/Succeeded",
          "level": "Info",
          "displayStatus": "Provisioning Succeeded",
          "message": ""
        }

      ],
      "statuses": [
        {
          "code": "ProvisioningState/Succeeded",
          "level": "Info",
          "displayStatus": "Provisioning Succeeded",
          "message": "Finished executing command"
        }

      ]

    }

  ],

  "statuses": [

    {
      "code": "ProvisioningState/Succeeded/osProvisioningComplete",
      "level": "Info",
      "displayStatus": "OS provisioning complete",
      "time": "2018-11-16T03:56:59.1063064+00:00"
    },

    {
      "code": "PowerState/running",
      "level": "Info",
      "displayStatus": "VM running"

    }

  ]

}

[6AC4:86EC][2018-11-16T05:39:57]: [Websites]: ****TryLogAppServiceResourceStatus Ends****
[6AC4:5F0C][2018-11-16T05:39:58]: [Websites]: System.AggregateException: One or more errors occurred. ---> System.Exception: **Deployment Failed. Refer deployment logs for more details**
   at Microsoft.Web.Hosting.SingleInstaller.Logic.SilentDeployLogic.<Run>d__a.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Microsoft.Web.Hosting.SingleInstaller.WixBA.ProcessCommandLine()
   at Microsoft.Web.Hosting.SingleInstaller.WixBA.Run()
---> (Inner Exception #0) System.Exception: Deployment Failed. Refer deployment logs for more details
   at Microsoft.Web.Hosting.SingleInstaller.Logic.SilentDeployLogic.<Run>d__a.MoveNext()<--- 

[6AC4:8418][2018-11-16T05:39:58]: Shutting down, exit code: 0xffffffff
[6AC4:8418][2018-11-16T05:39:58]: Variable: LocalAppDataFolder = C:\Users\AzureStackAdmin\AppData\Local\
[6AC4:8418][2018-11-16T05:39:58]: Variable: NETFRAMEWORK45 = 394802
[6AC4:8418][2018-11-16T05:39:58]: Variable: WixBundleAction = 4
[6AC4:8418][2018-11-16T05:39:58]: Variable: WixBundleElevated = 1
[6AC4:8418][2018-11-16T05:39:58]: Variable: WixBundleLog = D:\ASDKpostdeploytemp\ASDK\appservice\AppServiceLog1116-011218.txt
[6AC4:8418][2018-11-16T05:39:58]: Variable: WixBundleName = Microsoft Azure App Service
[6AC4:8418][2018-11-16T05:39:58]: Variable: WixBundleOriginalSource = D:\ASDKpostdeploytemp\ASDK\appservice\appservice.exe
[6AC4:8418][2018-11-16T05:39:58]: Variable: WixBundleProviderKey = {aecf39f0-59a8-4979-aa8b-585afd87273b}
[6AC4:8418][2018-11-16T05:39:58]: Variable: WixBundleTag =
[6AC4:8418][2018-11-16T05:39:58]: Variable: WixBundleVersion = 78.0.5.0
[6AC4:8418][2018-11-16T05:39:58]: Exit code: 0xffffffff, restarting: No
mattmcspirit commented 5 years ago

Do you have the full AppService log file that you could email me?

asdkconfigurator @ outlook . com - I’m curious if there’s anything else hidden in there.

Is the environment internet connected without a proxy or anything like that?

Thanks!

gijs007 commented 5 years ago

I've send the log files.

The server is hosted in a datacenter and has a corporate firewall in front of it. I haven't had any issues with the normal online deployment method, so I doubt that is causing any issues.

gijs007 commented 5 years ago

Note that: I'm not sure if this is a structural error. It could very well be that a redeploy would work fine, if issue #59 is resolved.

In that case, there should simply be some sort of retry for the VM deployment.

mattmcspirit commented 5 years ago

OK, after reviewing your logs, I don't think your original issue was due to the database. WHile there are many points in the large log file that say "Waiting for Hosting database to be ready", on all occasions, eventually, that was successful.

The error you have is around provisioning of CN0-VM, specifically "failed to provision vm extensions for vm cn0-vm" which i think may have been a random transient issue. I'm not sure how the App Service handles reinstalling over a partially failed attempt, so what I may do is write some cleanup of the resource group itself, and if there has been a failed run previously, I just completely clean the App Service resource group (and the SQL DB cleanup) and then it tries a fresh run again.

I'll work on that later.

mattmcspirit commented 5 years ago

So my challenge is, I don't know how the AppService.exe handles the re-run. We know it fails almost immediately if the database exists, and so we have accounted for that by cleaning the DB. But, if you look in the appservice-infra resource group, and click deployments, you'll see that there are 9 separate deployments that take place:

AppService.DeployCloud SharedWorkerTierScaleSetDeploy LargeWorkerTierScaleSetDeploy SmallWorkerTierScaleSetDeploy MediumWorkerTierScaleSetDeploy AppService.DeployTenantHydrationStorage AppService.DeployAdminHydrationStorage AppService.DeployUsageStorage AppService.DeployStorage

I don't know at what stage, any of these deployments write to the DB. In a default deployment on the ASDK, 7 of the 9 deployments listed above (from LargeWorker down to DeployStorage) finish in less than 2 minutes. If all of those succeed, but say, the AppService.DeployCloud fails, if we're cleaning up the database before the next attempt, will that break anything associated with the steps that had previously completed? That's the question I don't know.

It may be 'safer' and more predictable, to completely delete the appservice-infra resource group if it already exists, and then start a fresh for each attempt. I'll look into it a bit more though.

mattmcspirit commented 5 years ago

Let's retest with the latest 1809.2 version of the DeployAppService.ps1 - as mentioned here: https://github.com/mattmcspirit/azurestack/issues/59#issuecomment-439580037 - this should clean up failed runs and retry from the start again.

gijs007 commented 5 years ago

Unfortunately it still fails with the new script. I've send the log to your email address.

mattmcspirit commented 5 years ago

Yep, looking at your logs, my script is not catching the fact that you have a failed deployment within the appservice-infra resource group. Can you confirm that you still see the appservice-infra RG and that there is a failed deployment within it?

I'll work on adjusting the code to make it work.

mattmcspirit commented 5 years ago

Could you run this in your environment?

# Check if there is a previous failure for the App Service deployment - easier to completely clean the RG and start fresh
# Login to Azure Stack first
$azsLocation = (Get-AzsLocation).Name
$ArmEndpoint = "https://adminmanagement.local.azurestack.external"
Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "$ArmEndpoint" -ErrorAction Stop
Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" -ErrorAction Stop

$appServiceFailCheck = (Get-AzureRmResourceGroupDeployment -ResourceGroupName "appservice-infra" -Name "AppService.DeployCloud" -ErrorAction Stop)
$appServiceFailCheck.ProvisioningState

if ($($appServiceFailCheck.ProvisioningState) -eq 'Failed') {
    Write-Host "There is evidence of a previously failed App Service deployment in the App Service Resource Group. Starting cleanup..."
}

and post the output here? Does it print the 'There is evidence...' message?

gijs007 commented 5 years ago

The appservice-infra resource group doesn't exist. I do see appservice-fileshare and appservice-sql resource groups.

I just ran the commands you send, but it doesn't seem to print. The variable $appServiceFailCheck.ProvisioningState doesn't seem to be set.

PS C:\Windows\system32> # Check if there is a previous failure for the App Service deployment - easier to completely cle
an the RG and start fresh
PS C:\Windows\system32> # Login to Azure Stack first
PS C:\Windows\system32> $azsLocation = (Get-AzsLocation).Name
WARNING: Preview version of the module Azs.Subscriptions.Admin loaded. Future release of this module may have breaking
changes.
New-ServiceClient : Run Login-AzureRmAccount to login.
At C:\Program Files\WindowsPowerShell\Modules\Azs.Subscriptions.Admin\0.3.0\Generated.PowerShell.Commands\SwaggerPathCo
mmands\Get-AzsLocation.ps1:61 char:33
+ ... SubscriptionsAdminClient = New-ServiceClient @NewServiceClient_params
+                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : AzureRmContextError,New-ServiceClient

The property 'Locations' cannot be found on this object. Verify that the property exists.
At C:\Program Files\WindowsPowerShell\Modules\Azs.Subscriptions.Admin\0.3.0\Generated.PowerShell.Commands\SwaggerPathCo
mmands\Get-AzsLocation.ps1:66 char:9
+         $TaskResult = $SubscriptionsAdminClient.Locations.ListWithHtt ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], PropertyNotFoundException
    + FullyQualifiedErrorId : PropertyNotFoundStrict

PS C:\Windows\system32> $ArmEndpoint = "https://adminmanagement.local.azurestack.external"
PS C:\Windows\system32> Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "$ArmEndpoint" -ErrorAction Stop

Name                                              : AzureStackAdmin
EnableAdfsAuthentication                          : False
OnPremise                                         : False
ActiveDirectoryServiceEndpointResourceId          : https://adminmanagement.XXXX.onmicrosoft.com/XXXX
AdTenant                                          :
GalleryUrl                                        : https://adminportal.local.azurestack.external:30015/
ManagementPortalUrl                               :
ServiceManagementUrl                              :
PublishSettingsFileUrl                            :
ResourceManagerUrl                                : https://adminmanagement.local.azurestack.external
SqlDatabaseDnsSuffix                              :
StorageEndpointSuffix                             : local.azurestack.external
ActiveDirectoryAuthority                          : https://login.microsoftonline.com/
GraphUrl                                          : https://graph.windows.net/
GraphEndpointResourceId                           : https://graph.windows.net/
TrafficManagerDnsSuffix                           :
AzureKeyVaultDnsSuffix                            : vault.local.azurestack.external
DataLakeEndpointResourceId                        :
AzureDataLakeStoreFileSystemEndpointSuffix        :
AzureDataLakeAnalyticsCatalogAndJobEndpointSuffix :
AzureKeyVaultServiceEndpointResourceId            : https://vault.local.azurestack.external
AzureOperationalInsightsEndpointResourceId        :
AzureOperationalInsightsEndpoint                  :
VersionProfiles                                   : {}
ExtendedProperties                                : {}
BatchEndpointResourceId                           :

PS C:\Windows\system32> Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" -ErrorAction Stop

Account          : XXXXX@XXXXX.onmicrosoft.com
SubscriptionName : Default Provider Subscription
SubscriptionId   : XXXX
TenantId         : XXXX
Environment      : AzureStackAdmin

PS C:\Windows\system32>
PS C:\Windows\system32> $appServiceFailCheck = (Get-AzureRmResourceGroupDeployment -ResourceGroupName "appservice-infra"
 -Name "AppService.DeployCloud" -ErrorAction Stop)
Get-AzureRmResourceGroupDeployment : Resource group 'appservice-infra' could not be found.
At line:1 char:25
+ ... ailCheck = (Get-AzureRmResourceGroupDeployment -ResourceGroupName "ap ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Get-AzureRmResourceGroupDeployment], CloudException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.GetAzureResourceGroupDep
   loymentCmdlet

PS C:\Windows\system32> $appServiceFailCheck.ProvisioningState
PS C:\Windows\system32>
PS C:\Windows\system32> if ($($appServiceFailCheck.ProvisioningState) -eq 'Failed') {
>>     Write-Host "There is evidence of a previously failed App Service deployment in the App Service Resource Group. St
arting cleanup..."
>> }
PS C:\Windows\system32> if ($($appServiceFailCheck.ProvisioningState) -eq 'Failed') {
>>     Write-Host "There is evidence of a previously failed App Service deployment in the App Service Resource Group. Starting clean
up..."}
PS C:\Windows\system32> $appServiceFailCheck.ProvisioningState)
At line:1 char:39
+ $appServiceFailCheck.ProvisioningState)
+                                       ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : UnexpectedToken

PS C:\Windows\system32> $appServiceFailCheck.ProvisioningState
mattmcspirit commented 5 years ago

OK, this is strange:

PS C:\Windows\system32> $appServiceFailCheck = (Get-AzureRmResourceGroupDeployment -ResourceGroupName "appservice-infra"
 -Name "AppService.DeployCloud" -ErrorAction Stop)
Get-AzureRmResourceGroupDeployment : Resource group 'appservice-infra' could not be found.

If you look in the admin portal, under Resource Groups, is there one that shows "appservice-######"?

What is the result when you run:

Get-AzureRmResourceGroup
gijs007 commented 5 years ago

I don't have a resource group which is literally called "appservice-######" As mentioned earlier, I do have the appservice-fileshare and appservice-sql resource groups.

Output from Get-AzureRmResourceGroup:

ResourceGroupName : azurestack
Location          : westcentralus
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/azurestack

ResourceGroupName : cloud-shell-storage-westeurope
Location          : westeurope
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/cloud-shell-storage-westeurope

ResourceGroupName : RDGW
Location          : westeurope
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/RDGW
mattmcspirit commented 5 years ago

The output above, is from Azure. You need to make sure you're logged into Azure Stack first, then run Get-AzureRmResourceGroup.

So do this:

$azsLocation = (Get-AzsLocation).Name
$ArmEndpoint = "https://adminmanagement.local.azurestack.external"
Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "$ArmEndpoint" -ErrorAction Stop
Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" -ErrorAction Stop
Get-AzureRmResourceGroup

And see what comes back.

I'm curious why you don't have the appservice-infra, and if there's another random one that's been created. Aside from the appservice-sql and appservice-fileshare (which my script creates for you), if you have no other appservice- here, my 1809.2 script must have cleaned it up successfully, so now we have to figure out why your rerun is failing.

gijs007 commented 5 years ago

The output from that command is:

PS C:\Windows\system32> $azsLocation = (Get-AzsLocation).Name
WARNING: Preview version of the module Azs.Subscriptions.Admin loaded. Future release of this module may have breaking
changes.
Get-Exception : Code = InvalidResourceNamespace
Message = The resource namespace 'Microsoft.Subscriptions.Admin' is invalid.
At C:\Program Files\WindowsPowerShell\Modules\Azs.Subscriptions.Admin\0.3.0\Get-TaskResult.ps1:99 char:21
+                     Get-Exception -Exception $ex
+                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], Exception
    + FullyQualifiedErrorId : System.Exception,Get-Exception

PS C:\Windows\system32> $ArmEndpoint = "https://adminmanagement.local.azurestack.external"
PS C:\Windows\system32> Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "$ArmEndpoint" -ErrorAction Stop

Name                                              : AzureStackAdmin
EnableAdfsAuthentication                          : False
OnPremise                                         : False
ActiveDirectoryServiceEndpointResourceId          : https://adminmanagement.masdemo.onmicrosoft.com/XXXX
AdTenant                                          :
GalleryUrl                                        : https://adminportal.local.azurestack.external:30015/
ManagementPortalUrl                               :
ServiceManagementUrl                              :
PublishSettingsFileUrl                            :
ResourceManagerUrl                                : https://adminmanagement.local.azurestack.external
SqlDatabaseDnsSuffix                              :
StorageEndpointSuffix                             : local.azurestack.external
ActiveDirectoryAuthority                          : https://login.microsoftonline.com/
GraphUrl                                          : https://graph.windows.net/
GraphEndpointResourceId                           : https://graph.windows.net/
TrafficManagerDnsSuffix                           :
AzureKeyVaultDnsSuffix                            : vault.local.azurestack.external
DataLakeEndpointResourceId                        :
AzureDataLakeStoreFileSystemEndpointSuffix        :
AzureDataLakeAnalyticsCatalogAndJobEndpointSuffix :
AzureKeyVaultServiceEndpointResourceId            : https://vault.local.azurestack.external
AzureOperationalInsightsEndpointResourceId        :
AzureOperationalInsightsEndpoint                  :
VersionProfiles                                   : {}
ExtendedProperties                                : {}
BatchEndpointResourceId                           :

PS C:\Windows\system32> Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" -ErrorAction Stop

Account          : XXXX@XXXX.onmicrosoft.com
SubscriptionName : Default Provider Subscription
SubscriptionId   : XXXX
TenantId         : XXXX
Environment      : AzureStackAdmin

PS C:\Windows\system32> Get-AzureRmResourceGroup

ResourceGroupName : appservice-fileshare
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/appservice-fileshare

ResourceGroupName : appservice-sql
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/appservice-sql

ResourceGroupName : azurestack-activation
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/azurestack-activation

ResourceGroupName : azurestack-dbhosting
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/azurestack-dbhosting

ResourceGroupName : azurestack-images
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/azurestack-images

ResourceGroupName : system.keyvault
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/system.keyvault

ResourceGroupName : system.local
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/system.local

ResourceGroupName : system.local.adminkeyvault
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/system.local.adminkeyvault

ResourceGroupName : system.local.AzureMonitor
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/system.local.AzureMonitor

ResourceGroupName : system.local.dbadapter.dns
Location          : local
ProvisioningState : Succeeded
Tags              :
                    Name          Value
                    ============  ====================
                    ProviderName  SQLAdapter
                    Version       1.1.24.0
                    Category      Foundation
                    Role          DNSZoneResourceGroup

ResourceId        : /subscriptions/XXXX/resourceGroups/system.local.dbadapter.dns

ResourceGroupName : system.local.keyvault
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/system.local.keyvault

ResourceGroupName : system.local.MDM
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/system.local.MDM

ResourceGroupName : system.local.sqladapter
Location          : local
ProvisioningState : Succeeded
Tags              :
                    Name          Value
                    ============  =============
                    NextVersion
                    Version       1.1.24.0
                    Category      Foundation
                    Role          ResourceGroup
                    LastVersion
                    ProviderName  SQLAdapter

ResourceId        : /subscriptions/XXXX/resourceGroups/system.local.sqladapter

ResourceGroupName : System.UsageConnection
Location          : local
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/XXXX/resourceGroups/System.UsageConnection
mattmcspirit commented 5 years ago

OK, so that shows you have no left-over appservice-infra stuff, so there shouldn't be a problem re-deploying the App Service.

As a test, i crashed the App Service deployment within my new ASDK. Within the appservice-infra resource group, mine clearly showed 'failed' for the AppService.DeployCloud deployment. I then reran the script. My script successfully cleaned up the resuorce group, and successfully redeployed the App Service, so it must be something in your environment.

Can you try one more time, ensuring you've downloaded this version: https://raw.githubusercontent.com/mattmcspirit/azurestack/1809.2/deployment/powershell/DeployAppService.ps1 and copied it into your ConfigASDK\Scripts folder, overwriting the one that's there.

Then run the main script and see if it fails.

NOTE on your launch script, enclose your passwords in single quotes 'P@ssw0rd!' rather than double-quotes.

If it does fail, can you please email me the log file from the app service itself (not the log files in the ConfigASDK\Logs folder). This appservice log file will be within the downloadPath\ASDK\appservice folder. The name of the log will be something like AppServiceLog1118-145337 but your numbers will be different.

If it still fails with the 'network password' issue, we'll do some additional cleanup and try again.

gijs007 commented 5 years ago

I've send you the logs, as it still fails with the 'network password' issue. I've tried enclosing the passwords in single and double-quotes, but this makes no difference.

mattmcspirit commented 5 years ago

OK, here's what we're going to do.

  1. Navigate to C:\Windows\Temp\ on your ASDK host, and delete the Websites folder (this is where the App Service stores its downloaded files etc.
  2. Navigate to your downloadPath\ASDK\appservice folder and delete the following files:
    • 4 certificate files (PFX) and the AzureStackCertificationAuthority root cert.
    • AppServiceDeploymentSettings.json file, and the AppServicePreDeploymentSettings.json file.
    • AzureADPermissions and CertsCreated text files.
    • That should be a total of 9 files deleted.
  3. Login to the Azure portal (not Stack) in your web browser, and navigate to Azure Active Directory, then click on Enterprise Applications. In the main blade, under the reset button, in the search box, paste in: 31cc6ba0-2ef9-41e0-9536-f52ab79170fa and it should return a single result, called App Service. Click on 'App Service' then when the new blade opens, click the Delete button to delete that Azure AD App.
  4. Still in Azure Active Directory, click on App Registrations, paste in 31cc6ba0-2ef9-41e0-9536-f52ab79170fa in the search box, then use the drop down to change from 'My Apps' to 'All Apps'. It should display an App which you should be able to select and delete. If you can't delete it, click on Settings, Properties, then scroll down and change 'Multi Tenanted' to No, then delete it.

So, we've essentially cleaned up your temp folder, cleaned up certs and old txt files, and removed your app. We need to ensure that the script will run the 'AppServicePreReqs stage again, so we'll need to do this:

  1. Close all PowerShell windows and open an ISE as admin
  2. paste in the following:
$sqlServerInstance = '(localdb)\MSSQLLocalDB'
$databaseName = "ConfigASDK"
$tableName = "Progress"
$progressStage = "AddAppServicePreReqs"

Invoke-Sqlcmd -Server $sqlServerInstance -Query "USE $databaseName UPDATE Progress SET $progressStage = 'Incomplete';" -Verbose:$false -ErrorAction Stop
Read-SqlTableData -ServerInstance $sqlServerInstance -DatabaseName "$databaseName" -SchemaName "dbo" -TableName "$tableName" -ErrorAction SilentlyContinue -Verbose:$false

Then run it in the ISE. The output from the last command should show the AddAppServicePreReqs stage as 'Incomplete'.

Did that work? If so, close the ISE, ensure all PS windows are closed, then open a fresh PS Console/ISE and you should be good to try a rerun of the main script, and remember to use single quotes for containing passwords.

gijs007 commented 5 years ago

I followed all those steps, however the rerun didn't go very well:

errors

Shortly after it stopped running. Seems like it's not finding the script files, even thought they are there:

****** Please wait until all jobs have completed/failed before re-running the main script ******
At least one of the jobs failed.
FAILED JOB: Job Name: AddServerCoreImage | Error Message: The term '.\Scripts\AddImage.ps1' is not recognized as the nam
e of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, ve
rify that the path is correct and try again.
FAILED JOB: Job Name: AddServerFullImage | Error Message: The term '.\Scripts\AddImage.ps1' is not recognized as the nam
e of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, ve
rify that the path is correct and try again.
FAILED JOB: Job Name: AddMySQLAzpkg | Error Message: The term '.\Scripts\AddGalleryItems.ps1' is not recognized as the n
ame of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included,
verify that the path is correct and try again.
FAILED JOB: Job Name: AddSQLServerAzpkg | Error Message: The term '.\Scripts\AddGalleryItems.ps1' is not recognized as t
he name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was includ
ed, verify that the path is correct and try again.
FAILED JOB: Job Name: AddMySQLRP | Error Message: The term '.\Scripts\DeployDBRP.ps1' is not recognized as the name of a
 cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify t
hat the path is correct and try again.
FAILED JOB: Job Name: AddSQLServerRP | Error Message: The term '.\Scripts\DeployDBRP.ps1' is not recognized as the name
of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, veri
fy that the path is correct and try again.
FAILED JOB: Job Name: AddMySQLSku | Error Message: The term '.\Scripts\AddDBSkuQuota.ps1' is not recognized as the name
of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, veri
fy that the path is correct and try again.
FAILED JOB: Job Name: AddSQLServerSku | Error Message: The term '.\Scripts\AddDBSkuQuota.ps1' is not recognized as the n
ame of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included,
verify that the path is correct and try again.
FAILED JOB: Job Name: UploadScripts | Error Message: The term '.\Scripts\UploadScripts.ps1' is not recognized as the nam
e of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, ve
rify that the path is correct and try again.
FAILED JOB: Job Name: DeployMySQLHost | Error Message: The term '.\Scripts\DeployVM.ps1' is not recognized as the name o
f a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verif
y that the path is correct and try again.
FAILED JOB: Job Name: DeploySQLServerHost | Error Message: The term '.\Scripts\DeployVM.ps1' is not recognized as the na
me of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, v
erify that the path is correct and try again.
FAILED JOB: Job Name: DeployAppServiceFS | Error Message: The term '.\Scripts\DeployVM.ps1' is not recognized as the nam
e of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, ve
rify that the path is correct and try again.
FAILED JOB: Job Name: AddAppServicePreReqs | Error Message: The term '.\Scripts\AddAppServicePreReqs.ps1' is not recogni
zed as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path wa
s included, verify that the path is correct and try again.
FAILED JOB: Job Name: DeployAppService | Error Message: The term '.\Scripts\DeployAppService.ps1' is not recognized as t
he name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was includ
ed, verify that the path is correct and try again.
Please review the logs for further troubleshooting
At C:\ConfigASDK\Scripts\GetJobStatus.ps1:94 char:5
+     throw "Please review the logs for further troubleshooting"
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Please review t...troubleshooting:String) [], RuntimeException
    + FullyQualifiedErrorId : Please review the logs for further troubleshooting

PS C:\ConfigASDK>
mattmcspirit commented 5 years ago

This is bizarre, and I have no idea why that wouldn't work. There is no reason for that to fail.

gijs007 commented 5 years ago

Yes, that is all the case.

I'm retrying it again in a new PowerShell window again. Instead of using the CD command I used set-environment, I'm not sure if that matters.

Seems to work sort of better: errors2

gijs007 commented 5 years ago

It failed at the VM provisioning again, just like it did when I opened this issue.

mattmcspirit commented 5 years ago

Could you email the AppService log from within the ASDK\appservice folder? I don’t think think this is script related. Either something is wrong with the base Windows Server 2016 Datacenter image, or the virtualization stack. I’ll review the logs and if required, share with the App Service team.

gijs007 commented 5 years ago

I've send the log file.

mattmcspirit commented 5 years ago

OK, I've sent it to out internal App Service on Azure Stack team, to help us figure out what's happening here. I'll follow up when I have more info, but don't delete anything just yet, so we can investigate further. Thanks!

gijs007 commented 5 years ago

I think we can close this issue here? Since it also happens when installing App Services manually, so it's not script related.

Then again, you may want to keep it open to avoid other people from reporting the same issue ;)

mattmcspirit commented 5 years ago

Closed for now, we can always reopen later :)