search

mattmcspirit / azurestack

Azure Stack Resources
80 stars 41 forks source link

Error during Step SQLSERVERADDHOSTING #52

Closed stseifert closed 5 years ago

stseifert commented 6 years ago

I used ASDK Configurator 1808.1 in Offline Mode: Scenario 3 - Using Azure AD for authentication. You choose not to register the ASDK to Azure as part of the automated process. You have provided a valid -configAsdkOfflineZipPath

.\ConfigASDK.ps1 -azureDirectoryTenantName "*" -authenticationType AzureAD -downloadPath "D:\ASDKfiles" -configAsdkOfflineZipPath "C:\ASDKdependencies\ConfigASDKfiles.zip" -azureStackAdminPwd '****' -VMpwd '*' -azureAdUsername "" -azureAdPwd '' -skipMySQL

I got an error in step SQLSERVERADDHOSTING (see below). Any ideas ?

Stage Status

ExtractZip Complete InstallPowerShell Complete DownloadTools Complete HostConfiguration Complete Registration Skipped UbuntuImage Complete WindowsImage Complete MySQLGalleryItem Complete SQLServerGalleryItem Complete VMExtensions Skipped MySQLRP Complete SQLServerRP Complete MySQLSKUQuota Complete SQLServerSKUQuota Complete UploadScripts Complete MySQLDBVM Complete SQLServerDBVM Complete MySQLAddHosting Skipped SQLServerAddHosting Failed AppServiceFileServer Incomplete AppServiceSQLServer Incomplete DownloadAppService Incomplete GenerateAppServiceCerts Incomplete CreateServicePrincipal Incomplete GrantAzureADAppPermissions Incomplete InstallAppService Incomplete RegisterNewRPs Incomplete CreatePlansOffers Incomplete InstallHostApps Incomplete CreateOutput Incomplete

VERBOSE: [11:08 AM]::[SQLSERVERADDHOSTING]:: 11:08:18 AM - Resource Microsoft.SQLAdapter.Admin/hostingServers 'sqlhost.local.cloudapp.azurestack.external' failed with message '{ "error": { "code": "-2146232060", "message": "A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - The remote computer refused the network connection.)" } }'.Exception.Message

stseifert commented 6 years ago

Error occurs during following command ew-AzureRmResourceGroupDeployment -ResourceGroupName "azurestack-dbhosting" -TemplateUri $templateURI ` -hostingServerName $sqlFqdn -hostingServerSQLLoginName "sa" -hostingServerSQLLoginPassword $secureVMpwd -totalSpaceMB 20480 -skuName "MSSQL2017" -Mode Incremental -Verbose -ErrorAction Stop

mattmcspirit commented 6 years ago

Hey - thanks for the message.

So, what I think the error is, is that there are missing dependencies for the SQL Server 2017 deployment. If you log into the Azure Stack Admin portal, you'll find that in the azurestack-dbhosting resource group, the SQL Server VM has deployed correctly, but it's the final stage, 'CustomScriptExtension' which fails.

It is failing because the Bash script that installs SQL Server on Ubuntu is missing some .deb files. The error message: "message": "A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections" actually is telling me that the SQL Server service isn't actually running. I discovered this a few days ago and have fixed it in the next release of my script, which I'm currently testing. It will be released early next week.

To help unblock you for now, here's what I would suggest (This is the quickest way)

1) download this zip file that i just created for you, to your ASDK host: it contains all the up to date binaries for MSSQL on Ubuntu and updated versions of the install_MSSQL script - https://oelvhds.blob.core.windows.net/binaries/binaries.zip

2) Extract the ZIP file and then open the Azure Stack Admin portal, go to the "azurestack-offline" resource group, "offlinestor" storage account and the "offlinecontainer" container. In there, you should see all the existing binaries and .sh scripts. Delete them all, then upload all the new .deb files, and scripts from the zip file I provided.

3) Go to the "azurestack-dbhosting" resource group, and delete all "SQL" resources (not the MySQL ones)

4) Open your ConfigAsdkProgress.csv file (in your ASDKfiles folder) - edit the line SQLServerDBVM Complete to read SQLServerDBVM Incomplete, then save it. You may need to save it to your desktop (ensure you save as a .CSV and not .txt) and then copy it back to it's original location in your ASDKfiles folder.

5) Close all PowerShell windows, then re-open an admin PowerShell console/ISE and re-run your launch script:

.\ConfigASDK.ps1 -azureDirectoryTenantName "vwfsdev.onmicrosoft.com" -authenticationType AzureAD -downloadPath "D:\ASDKfiles" -configAsdkOfflineZipPath "C:\ASDKdependencies\ConfigASDKfiles.zip" -azureStackAdminPwd '****'
-VMpwd '*******' -azureAdUsername "" -azureAdPwd '' -skipMySQL

The script should re-deploy the SQL Server and use the update bash script to grab the correctd dependencies and install SQL Server.

Let me know if that works.

Thanks! Matt

stseifert commented 6 years ago

Hi Matt,

thank’s for your quick response. Will try it and let you know…

Freundliche Grüße / Kind regards Steffen Seifert

IH-IOPI – .NET Internetsystems and Platform Operations

Volkswagen Financial Services Digital Solutions GmbH Brieffach / Postbox: IH-IOPI Schmalbachstraße 1 D-38112 Braunschweig Tel.: +49 531-212-4158 Fax: +49 531-212-74158 email: steffen.seifert@vwfs.commailto:steffen.seifert@vwfs.com

Von: Matt McSpirit notifications@github.com Gesendet: Donnerstag, 1. November 2018 18:51 An: mattmcspirit/azurestack azurestack@noreply.github.com Cc: Seifert, Steffen steffen.seifert@vwfs.com; Author author@noreply.github.com Betreff: Re: [mattmcspirit/azurestack] Error during Step SQLSERVERADDHOSTING (#52)

Hey - thanks for the message.

So, what I think the error is, is that there are missing dependencies for the SQL Server 2017 deployment. If you log into the Azure Stack Admin portal, you'll find that in the azurestack-dbhosting resource group, the SQL Server VM has deployed correctly, but it's the final stage, 'CustomScriptExtension' which fails.

It is failing because the Bash script that installs SQL Server on Ubuntu is missing some .deb files. The error message: "message": "A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections" actually is telling me that the SQL Server service isn't actually running. I discovered this a few days ago and have fixed it in the next release of my script, which I'm currently testing. It will be released early next week.

To help unblock you for now, here's what I would suggest (This is the quickest way)

  1. download this zip file that i just created for you, to your ASDK host: it contains all the up to date binaries for MSSQL on Ubuntu and updated versions of the install_MSSQL script - https://oelvhds.blob.core.windows.net/binaries/binaries.zip

  2. Extract the ZIP file and then open the Azure Stack Admin portal, go to the "azurestack-offline" resource group, "offlinestor" storage account and the "offlinecontainer" container. In there, you should see all the existing binaries and .sh scripts. Delete them all, then upload all the new .deb files, and scripts from the zip file I provided.

  3. Go to the "azurestack-dbhosting" resource group, and delete all "SQL" resources (not the MySQL ones)

  4. Open your ConfigAsdkProgress.csv file (in your ASDKfiles folder) - edit the line SQLServerDBVM Complete to read SQLServerDBVM Incomplete, then save it. You may need to save it to your desktop (ensure you save as a .CSV and not .txt) and then copy it back to it's original location in your ASDKfiles folder.

  5. Close all PowerShell windows, then re-open an admin PowerShell console/ISE and re-run your launch script:

.\ConfigASDK.ps1 -azureDirectoryTenantName "vwfsdev.onmicrosoft.com" -authenticationType AzureAD -downloadPath "D:\ASDKfiles" -configAsdkOfflineZipPath "C:\ASDKdependencies\ConfigASDKfiles.zip" -azureStackAdminPwd '****'

-VMpwd '***' -azureAdUsername "" -azureAdPwd '' -skipMySQL

The script should re-deploy the SQL Server and use the update bash script to grab the correctd dependencies and install SQL Server.

Let me know if that works.

Thanks! Matt

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/mattmcspirit/azurestack/issues/52#issuecomment-435123027, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AU--FmC-PfMy84A_kBZdlPOXmoz3s2t2ks5uqzRpgaJpZM4YGaPu.

Volkswagen Financial Services Digital Solutions GmbH Sitz/Registered seat: Braunschweig Registergericht/Registration court: Amtsgericht Braunschweig HRB Nr./Commercial Register No.: 201153 Geschäftsführer/Executive Director: Torsten Zibell (Sprecher/Spokesman), Jens Springer, Thomas Sutter

Wichtiger Hinweis: Die vorgenannten Angaben werden jeder E-Mail automatisch hinzugefügt und lassen keine Rückschlüsse auf den Rechtscharakter der E-Mail zu. Important note: The above information is automatically added to this e-mail. This addition does not constitute a representation that the content of this e-mail is legally relevant and/or is intended to be legally binding upon Volkswagen Financial Services Digital Solutions GmbH.

mattmcspirit commented 6 years ago

Hey Steffen,

I think you are correct. My new version, 1809, fixes all of this, however i wouldn't run the 1809 version on a system where you have already run the 1808.1, so we'll try and fix this for now.

The $offlineBaseURI is first defined on line 3064:

$offlineBaseURI = ('{0}{1}/' -f $asdkOfflineStorageAccount.PrimaryEndpoints.Blob.AbsoluteUri, $asdkOfflineContainerName) -replace "https", "http"

So, let's test why it's not getting populated. Try this:

Login-AzureRmAccount -EnvironmentName "AzureStackAdmin"
$asdkOfflineRGName = "azurestack-offline"
$asdkOfflineStorageAccountName = "offlinestor"
$asdkOfflineContainerName = "offlinecontainer" 
$asdkOfflineStorageAccount = Get-AzureRmStorageAccount -Name $asdkOfflineStorageAccountName -ResourceGroupName $asdkOfflineRGName
Set-AzureRmCurrentStorageAccount -StorageAccountName $asdkOfflineStorageAccountName -ResourceGroupName $asdkOfflineRGName
$asdkOfflineContainer = Get-AzureStorageContainer -Name $asdkOfflineContainerName
$offlineBaseURI = ('{0}{1}/' -f $asdkOfflineStorageAccount.PrimaryEndpoints.Blob.AbsoluteUri, $asdkOfflineContainerName) -replace "https", "http"
$offlineBaseURI

if $offlineBaseURI is still empty, try this:

$offlineBaseURI = ('{0}{1}/' -f $asdkOfflineStorageAccount.PrimaryEndpoints.Blob, $asdkOfflineContainerName) -replace "https", "http"
$offlineBaseURI

Let me know what the results are.

Thanks! Matt

mattmcspirit commented 6 years ago

In addition, i think i know why the $offlineBaseURI is not getting populated.

Based on your ConfigASDKprogressLog.csv, you show that the 'UploadScripts' step is complete. The $offlineBaseURI is created and set in this section. So, when you rerun the script at a later time, and 'UploadScripts' step is already completed, my script skips the completed step, and therefore, it doesn't set $offlineBaseURI again, so it's left empty.

Here's a fix to try when you've tested the stuff above - paste this into the ConfigASDK script at line 3013 and replace what's currently there. 

$progress = Import-Csv -Path $ConfigASDKProgressLogPath
$RowIndex = [array]::IndexOf($progress.Stage, "UploadScripts")
$scriptStep = $($progress[$RowIndex].Stage).ToString().ToUpper()
if ($progress[$RowIndex].Status -eq "Complete") {
    Write-CustomVerbose -Message "ASDK Configuration Stage: $($progress[$RowIndex].Stage) previously completed successfully"
    if ($deploymentMode -ne "Online") {
        $asdkOfflineRGName = "azurestack-offline"
        $asdkOfflineStorageAccountName = "offlinestor"
        $asdkOfflineContainerName = "offlinecontainer"
        Login-AzureRmAccount -EnvironmentName "AzureStackAdmin" -TenantId $TenantID -Credential $asdkCreds -ErrorAction Stop | Out-Null
        $asdkOfflineStorageAccount = Get-AzureRmStorageAccount -Name $asdkOfflineStorageAccountName -ResourceGroupName $asdkOfflineRGName -ErrorAction SilentlyContinue
        Set-AzureRmCurrentStorageAccount -StorageAccountName $asdkOfflineStorageAccountName -ResourceGroupName $asdkOfflineRGName
        $asdkOfflineContainer = Get-AzureStorageContainer -Name $asdkOfflineContainerName -ErrorAction SilentlyContinue
        $offlineBaseURI = ('{0}{1}/' -f $asdkOfflineStorageAccount.PrimaryEndpoints.Blob.AbsoluteUri, $asdkOfflineContainerName) -replace "https", "http"
    }
}

That should help fix things i think.

mattmcspirit commented 6 years ago

Hey,

I think the only things left to check would be inside the sqlhost VM - I would Putty into the VM (sqladmin, and your VM password), then sudo -s, then check if the SQL Server service is active and running - systemctl status mssql-server. If that's not running, or installed at all, then that is your issue.

Alternatively, you could try to test this step manually by logging into the Azure Stack admin portal and going to SQL Adapter, and trying to add a hosting server manually, using sqlhost.cloudapp.local.azurestack.external, and then your sa and VM password credentials.

If none of those things work, we should check the logs for the CustomScriptExtension inside the sqlhost VM, as that will help us understand why the SQL Server install failed.

I'm likely to release my 1809 version tomorrow, which i know works for offline/partial scenarios as i've been testing it over the last few days, so it may be quicker to wipe and reload your ASDK and use this version (you would need to download a new ConfigASDKFiles.zip as there are some new things in there.

Hope that helps! Matt

mattmcspirit commented 6 years ago

This is really strange - if the SQL Server service is running, i believe that the Linux bash script completed successfully, but we can take a look here: https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linux#troubleshooting

sudo -s to elevate yourself, then navigate to the folder here /var/lib/waagent/custom-script/download/0/, and once there, you should be able to open log files using nano .

Do the same here also: https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linuxostc#troubleshooting as there are a couple of extra log files created in that folder.

One thing you could try is to install SQL Server Management Studio on the ASDK host and see if you can connect to the SQL Server through management studio. If you can, we then know it's an issue with Azure Stack and the SQL RP, rather than the SQLHOST VM itself. If you can't connect with SQL Management Studio, it could be firewall within the SQLHOST VM, or something else blocking.

Hope that helps, Matt

mattmcspirit commented 6 years ago

Hi Steffen,

So when I publish a new version of the ConfigASDK.ps1 script, I also publish a new version of ConfigASDKdependencies.ps1.

Did you download the new ConfigASDK.ps1 but use an old ConfigASDKdependencies.ps1 file to create your ZIP?

I've just tested the ConfigASDKdependencies.ps1 script here: https://raw.githubusercontent.com/mattmcspirit/azurestack/master/deployment/offline/ConfigASDKdependencies.ps1 and it's downloading all files without issue.

Thanks, Matt

mattmcspirit commented 5 years ago

Let me know if you're still encountering issues, else, I'll close this one out.

Thanks!

stseifert commented 5 years ago

Hi Matt,

Unfortunately we still have problems with the 1809 scripts. The access to the internet from DevKit is only possible via dedicated (non-transparent) proxy. Heyko also tested the installation on site last week and made some modifications to the script. Does the installation of AppServices and resource providers in partial online mode really need a connection to the internet? Do you have any ideas how we can get ahead?

My goal is to offer our developers a monthly Devkit with the latest features. Therefore we have to automate the installation.

F.y.i.: We are nominated for the ADSK Proxy Preview.

Freundliche Grüße / Kind regards Steffen Seifert

IH-IOPI – .NET Internetsystems and Platform Operations

Volkswagen Financial Services Digital Solutions GmbH Brieffach / Postbox: IH-IOPI Schmalbachstraße 1 D-38112 Braunschweig Tel.: +49 531-212-4158 Fax: +49 531-212-74158 email: steffen.seifert@vwfs.commailto:steffen.seifert@vwfs.com

Von: Matt McSpirit notifications@github.com Gesendet: Montag, 26. November 2018 20:50 An: mattmcspirit/azurestack azurestack@noreply.github.com Cc: Seifert, Steffen Steffen.Seifert@vwfs.com; Author author@noreply.github.com Betreff: Re: [mattmcspirit/azurestack] Error during Step SQLSERVERADDHOSTING (#52)

Let me know if you're still encountering issues, else, I'll close this one out.

Thanks!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/mattmcspirit/azurestack/issues/52#issuecomment-441774248, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AU--Fu6nkP45xfhNG3u_98VjuFmtcSEqks5uzEYEgaJpZM4YGaPu.

Volkswagen Financial Services Digital Solutions GmbH Sitz/Registered seat: Braunschweig Registergericht/Registration court: Amtsgericht Braunschweig HRB Nr./Commercial Register No.: 201153 Geschäftsführer/Executive Director: Torsten Zibell (Sprecher/Spokesman), Jens Springer, Thomas Sutter

Wichtiger Hinweis: Die vorgenannten Angaben werden jeder E-Mail automatisch hinzugefügt und lassen keine Rückschlüsse auf den Rechtscharakter der E-Mail zu. Important note: The above information is automatically added to this e-mail. This addition does not constitute a representation that the content of this e-mail is legally relevant and/or is intended to be legally binding upon Volkswagen Financial Services Digital Solutions GmbH.