search

Esri / arcgis-powershell-dsc

This repository contains scripts, code and samples for automating the install and configuration of ArcGIS (Enterprise and Desktop) using Microsoft Windows PowerShell DSC (Desired State Configuration).
Apache License 2.0
113 stars 61 forks source link

HA Datastore upgrade to 10.9 fails when "EnableFailoverOnPrimaryStop" is set to true #325

Closed Biboba closed 2 years ago

Biboba commented 3 years ago

Community Note

Module Version

Affected Resource(s)

Configuration Files

ArcGIS_Enterprise_HA.txt

Expected Behavior

Upgrading datastore should be successful.

Actual Behavior

Upgrading datastore seems to be successful (no error thrown) but the datastore is actually not upgraded. Rerunning the configuration does not solve the issue.

Steps to Reproduce

Upgrading from 10.8.1 to 10.9 with the config file above:

Invoke-ArcGISConfiguration -ConfigurationParametersFile C:\scripts\config\MapsQADeployment_HA.json -Credential (Get-Credential) -Mode Upgrade

Important Factoids

Log from DataStoreUpgradeConfigure on the primary datastore host:

20-May-21 2:36:56 PM: [MAPSPORTALQA01]: LCM:  [ Start  Set      ]
20-May-21 2:36:56 PM: [MAPSPORTALQA01]: LCM:  [ Start  Resource ]  [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade]
20-May-21 2:36:56 PM: [MAPSPORTALQA01]: LCM:  [ Start  Test     ]  [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade]
20-May-21 2:37:31 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] Response:- {"currentVersion":"10.9.0","relational":{"portalUrl":"https://mapsqa.company.com/geoportal","registered":true,"configureRole":"PRIMARY"},"tileCache":{"portalUrl":"https://mapsqa.company.com/geoportal","registered":true,"configureRole":"PRIMARY"},"upgrading":"inplace","defaultContentDirectory":"C:\\arcgis\\arcgisdatastore\\"}
20-May-21 2:37:31 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] Upgrade in progress - inplace
20-May-21 2:37:31 PM: [MAPSPORTALQA01]: LCM:  [ End    Test     ]  [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade]  in 36.7700 seconds.
20-May-21 2:37:31 PM: [MAPSPORTALQA01]: LCM:  [ Start  Set      ]  [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade]
20-May-21 2:37:31 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] Retrieved server token successfully
20-May-21 2:37:31 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] POST https://localhost:2443/arcgis/datastoreadmin/configure with 256-byte payload
20-May-21 2:37:51 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] received -1-byte response of content type application/json;charset=UTF-8
20-May-21 2:37:51 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] Response:- {"currentVersion":"10.9.0","relational":{"portalUrl":"https://mapsqa.company.com/geoportal","registered":true,"configureRole":"PRIMARY"},"tileCache":{"portalUrl":"https://mapsqa.company.com/geoportal","registered":true,"configureRole":"PRIMARY"},"spatioTemporal":{"configureRole":"BIGDATA_ONLY"},"upgrading":"inplace","defaultContentDirectory":"C:\\arcgis\\arcgisdatastore\\"}
20-May-21 2:37:51 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] Relational Replication Role - PRIMARY
20-May-21 2:37:51 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] TileCache Replication Role - PRIMARY
20-May-21 2:37:51 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] relational,tilecache
20-May-21 2:37:51 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] Executing C:\Program Files\Arcgis\DataStore\tools\configuredatastore.bat
20-May-21 2:38:11 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] Output of execution:- 
Configuring data store(s). Initial configuration may take a few minutes. When configuring an upgraded data store, this process may take several hours depending on the size of your data. Please wait...

Performing upgrade prerequisite checks on data stores...

Prerequisite checks were not met for 'relational' data store:
    MAPSPORTALQA02.company.com: Unable to perform prerequisite check. Please make sure that the latest software is installed, the machine is accessible, and ArcGIS Data Store service is started.

Prerequisite checks were not met for 'tile cache' data store:

    MAPSPORTALQA02.company.com: Unable to perform prerequisite check. Please make sure that the latest software is installed, the machine is accessible, and ArcGIS Data Store service is started.
20-May-21 2:38:11 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] Upgraded did not succeed. Process exit code:- 1
20-May-21 2:38:11 PM: [MAPSPORTALQA01]:                            [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade] 
Error encountered: Unable to perform prerequisite check. Please make sure that the latest software is installed, the machine is accessible, and ArcGIS Data Store service is started.
20-May-21 2:38:11 PM: [MAPSPORTALQA01]: LCM:  [ End    Set      ]  [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade]  in 41.7480 seconds.
20-May-21 2:38:11 PM: [MAPSPORTALQA01]: LCM:  [ End    Resource ]  [[ArcGIS_DataStoreUpgrade]DataStoreConfigUpgrade]
20-May-21 2:38:11 PM: [MAPSPORTALQA01]: LCM:  [ End    Set      ]
20-May-21 2:38:11 PM: [MAPSPORTALQA01]: LCM:  [ End    Set      ]    in  78.6040 seconds.
20-May-21 2:38:11 PM: Operation 'Invoke CimMethod' complete.

It fails because it is complaning that it cannot run the prerequisite check on the standby datastore. Going on the standby datastore host, I noticed that the service is stopped. The reason ? It has been stopped previously by DSC in the "DataStoreUpgradePreInstall" module.

Log from the standby datastore "DataStoreUpgradePreInstall":

20-May-21 2:36:37 PM: [MAPSPORTALQA02]: LCM:  [ Start  Set      ]
20-May-21 2:36:37 PM: [MAPSPORTALQA02]: LCM:  [ Start  Resource ]  [[Service]ArcGIS_DataStore_Service_Stop]
20-May-21 2:36:37 PM: [MAPSPORTALQA02]: LCM:  [ Start  Test     ]  [[Service]ArcGIS_DataStore_Service_Stop]
20-May-21 2:36:37 PM: [MAPSPORTALQA02]:                            [[Service]ArcGIS_DataStore_Service_Stop] Perform operation 'Query CimInstances' with following parameters, ''queryExpression' = SELECT * FROM Win32_Service WHERE Name='ArcGIS Data Store','queryDialect' = WQL,'namespaceName' = root\cimv2'.
20-May-21 2:36:37 PM: [MAPSPORTALQA02]:                            [[Service]ArcGIS_DataStore_Service_Stop] Operation 'Query CimInstances' complete.
20-May-21 2:36:37 PM: [MAPSPORTALQA02]: LCM:  [ End    Test     ]  [[Service]ArcGIS_DataStore_Service_Stop]  in 0.3080 seconds.
20-May-21 2:36:37 PM: [MAPSPORTALQA02]: LCM:  [ Start  Set      ]  [[Service]ArcGIS_DataStore_Service_Stop]
20-May-21 2:36:37 PM: [MAPSPORTALQA02]:                            [[Service]ArcGIS_DataStore_Service_Stop] Service 'ArcGIS Data Store' already exists. Write properties such as Status, DisplayName, Description, Dependencies will be ignored for existing services.
20-May-21 2:36:37 PM: [MAPSPORTALQA02]:                            [[Service]ArcGIS_DataStore_Service_Stop] Perform operation 'Query CimInstances' with following parameters, ''queryExpression' = SELECT * FROM Win32_Service WHERE Name='ArcGIS Data Store','queryDialect' = WQL,'namespaceName' = root\cimv2'.
20-May-21 2:36:37 PM: [MAPSPORTALQA02]:                            [[Service]ArcGIS_DataStore_Service_Stop] Operation 'Query CimInstances' complete.
20-May-21 2:36:37 PM: [MAPSPORTALQA02]:                            [[Service]ArcGIS_DataStore_Service_Stop] Service 'ArcGIS Data Store' stopped.
20-May-21 2:36:37 PM: [MAPSPORTALQA02]: LCM:  [ End    Set      ]  [[Service]ArcGIS_DataStore_Service_Stop]  in 2.4730 seconds.
20-May-21 2:36:37 PM: [MAPSPORTALQA02]: LCM:  [ End    Resource ]  [[Service]ArcGIS_DataStore_Service_Stop]
20-May-21 2:36:37 PM: [MAPSPORTALQA02]: LCM:  [ End    Set      ]
20-May-21 2:36:37 PM: [MAPSPORTALQA02]: LCM:  [ End    Set      ]    in  3.0680 seconds.
20-May-21 2:36:37 PM: Operation 'Invoke CimMethod' complete.

So I think there is something wrong with the upgrade workflow when "EnableFailoverOnPrimaryStop" is set to true which triggers the "DataStoreUpgradePreInstall":

https://github.com/Esri/arcgis-powershell-dsc/blob/78e488fc0aac6f8f4cdf574e1bdb50a7d39ed350/Modules/ArcGIS/ArcGIS.psm1#L2156-L2163

Thanks for your help and for this nice library :-)

Biboba commented 3 years ago

Hello @cameronkroeker, Were you able to reproduce this issue ? Any workaround ?

Thanks !

cameronkroeker commented 3 years ago

Hello @cameronkroeker, Were you able to reproduce this issue ? Any workaround ?

Thanks !

Hello @Biboba,

I tested similar scenario prior to release and did not see this issue. However, I will see if it can reproduced using your json config file.

Thanks, Cameron K.

cameronkroeker commented 3 years ago

Hi @Biboba,

I was able to reproduce this issue. When "EnableFailoverOnPrimaryStop" is set to true, the module will first stop the standby prior to installing so that a failover isn't invoked once the installation is complete (the setup will stop/start the service which may inadvertently invoke a failover especially if standby completes before primary).

We will work on getting this fixed in the next release, however in the meantime here is a workaround:

  1. Manually start Standby Data Store service in services.msc
  2. In your json config file change "EnableFailoverOnPrimaryStop" to false
  3. Run the upgrade via DSC -Mode Upgrade, this will upgrade the Data Store successfully
  4. Once the upgrade completes, verify\validate it
  5. In your json config file update "EnableFailoverOnPrimaryStop" back to true
  6. Run DSC with -Mode InstallLicenseConfigure, which will then re-enable EnableFailoverOnPrimaryStop.

Thanks, Cameron K.

Biboba commented 3 years ago

Hi @cameronkroeker,

Thanks for your reply. I confirm that following your workaround, it works.

Thanks !

cameronkroeker commented 2 years ago

Hello @Biboba,

This issue has been addressed in the latest release, v3.3.0:

https://github.com/Esri/arcgis-powershell-dsc/releases/tag/v3.3.0

Thanks, Cameron K.