search

Esri / arcgis-powershell-dsc

This repository contains scripts, code and samples for automating the install and configuration of ArcGIS (Enterprise and Desktop) using Microsoft Windows PowerShell DSC (Desired State Configuration).
Apache License 2.0
113 stars 61 forks source link

Executing an 'upgrade" from 10.7.1 to 10.8.1 has a Federated Server in a bad state: "Failed to Install Server" #301

Closed pfoppe closed 3 years ago

pfoppe commented 3 years ago

Community Note

Module Version

Affected Resource(s)

Configuration Files

Below you will find the contents of (2) Configuration Files - 1) The "Original" configuration file used to install the ArcGIS Server Site at v10.7.1 and federate with the portal 2) The "Upgrade" configuration file used to upgrade the site to 10.8.1 and retain portal federation

--- Original 10.7.1 "Install" config file ---

{
 "AllNodes": [
    {
      "NodeName": "<MACHINE1>",
      "Role": [
        "ServerWebAdaptor"
      ],
      "SslCertificates": [
        {
          "Path": "<\\server\\share\\file>",
          "Alias": "<ALIAS>",
          "Password": "<PASSWORD>",
          "Target": [
            "WebAdaptor"
          ]
        }
      ]   
    },
    {
      "NodeName": "<MACHINE2>",
      "Role": [
        "Server"
      ]
    }
  ],
  "ConfigData": {
    "Version": "10.7.1",
    "ServerContext": "arcgisfed",
    "ServerRole": "GeneralPurposeServer",
    "SevenZipInstallerPath": "<\\server\\share\\file>",
    "Credentials": {
      "PrimarySiteAdmin": {
        "UserName": "<REDACTED>",
        "Email": "<REDACTED>",
        "Password": "<REDACTED>"
      },
      "ServiceAccount": {
        "Password": "<REDACTED>",
        "UserName": "<DOMAIN\USERNAME>",
        "IsDomainAccount": true
      }
    },
    "Server": {
      "LicenseFilePath": "<\\server\\share\\file>",
      "Installer": {
        "Path": "<\\server\\share\\file>",
        "InstallDir": "C:\\Program Files\\ArcGIS\\Server",
        "InstallDirPython": "C:\\Python27",
        "PatchesDir": "<\\server\\share\\file>"
      },
      "ServerDirectoriesRootLocation": "<\\server\\share\\directories>",
      "ConfigStoreLocation": "<\\server\\share\\config-store>",
      "RegisteredDirectories": []
    },
    "WebAdaptor": {
      "AdminAccessEnabled": true,
      "Installer": {
        "Path": "<\\server\\share\\file>"
      }
    },
    "Federation": {
        "PortalHostName": "<ALIAS>",
        "PortalPort": "443",
        "PortalContext": "portal",
        "PrimarySiteAdmin": {
            "UserName": "<REDACTED>",
            "Password": "<REDACTED>"
        }
    }
  }
}

--- Upgrade to 10.8.1 config file ---

{
  "AllNodes": [
    {
      "NodeName": "<MACHINE1>",
      "Role": [
        "ServerWebAdaptor"
      ],
      "SslCertificates": [
        {
          "Path": "<\\server\\share\\file>",
          "CNameFQDN": "<ALIAS>",
          "Password": "<PASSWORD>",
          "Target": [
            "WebAdaptor"
          ]
        }
      ]
    },
    {
      "NodeName": "<MACHINE2>",
      "Role": [
        "Server"
      ]
    }
  ],
  "ConfigData": {
    "Version": "10.8.1",
    "OldVersion": "10.7.1",
    "ServerContext": "arcgisfed",
    "ServerRole": "GeneralPurposeServer",
    "Credentials": {
        "ServiceAccount": {
        "Password": "<REDACTED>",
        "UserName": "<DOMAIN\USERNAME>",
        "IsDomainAccount": true
      },
      "ADServiceUser": {
        "Password": "<REDACTED>",
        "UserName": "<DOMAIN\USERNAME>",
        "IsDomainAccount": true
      }
    },
    "Server": {
      "LicenseFilePath": "<\\server\\share\\file>",
      "Installer": {
        "Path": "<\\server\\share\\file>",
        "InstallDir": "C:\\Program Files\\ArcGIS\\Server",
        "InstallDirPython": "C:\\Python27",
        "PatchesDir": "<\\server\\share\\file>"
      },
     "ServerDirectoriesRootLocation": "<\\server\\share\\directories>",
     "ConfigStoreLocation": "<\\server\\share\\config-store>",
     "PrimarySiteAdmin": {
       "UserName": "<REDACTED>",
       "Email": "<REDACTED>",
       "Password": "<REDACTED>"
      }
    },
    "WebAdaptor": {
      "AdminAccessEnabled": true,
      "Installer": {
       "Path": "<\\server\\share\\file>"
      }
    },
    "Federation": {
      "PortalHostName": "<ALIAS>",
      "PortalPort": "443",
      "PortalContext": "portal",
      "PortalAdministrator": {
        "UserName": "<REDACTED>",
        "Password": "<REDACTED>"
      }
    }
  }
}

Expected Behavior

  1. MACHINE2 should have been upgraded to 10.8.1 (from 10.7.1)
  2. MACHINE1 should have a web-adaptor installed with the context "arcgisfed" and registered with MACHINE2
  3. MACHINE2 should stay federated with the portal hosted on and referenced via the Alias applied with the "CNameFQDN" property

Actual Behavior

Behavior 1 and 2 above failed Behavior 3 seem to have succeeded. The portal still has this ArcGIS Server site (with its alias) marked as a 'federated' server and items seem to still be in the portal

Steps to Reproduce

  1. Complete a PSDSC InstallLicenseConfigure for Esri Version 10.7.1 using PSDSC v2.xx
  2. Attempt an upgrade to Esri version 10.8.1 using PSDSC v3.1.1

Important Factoids

Running windows server 2016 on all hosts (PSDSC execution server, IIS/Portal/Web-Adaptor server, Federated ArcGIS Server) This is a 6 machine setup with (4) JSON files run in the following order - JSON File 1 - MACHINE1 with role Portal, ServerWebAdaptor,PortalWebAdaptor MACHINE3 with relational data store MACHINE4 with role "Server" and designated as the hosting server (web-adaptor on

JSON File 2 - NOTE - These are the JSON Files above. I did not provide other JSON files MACHINE1 - Hosts the server web-adaptor for this installation (federated server) MACHINE2 - "Federated" ArcGIS Server (no hosting server, no data store, etc)

JSON File 3 - MACHINE1 - Hosts the server web-adaptor for this installation (federated server setup for Raster Analysis) MACHINE5 - "Federated" ArcGIS Server setup as the Raster Analysis server

JSON File 4 - MACHINE1 - Hosts the server web-adaptor for this installation (federated server setup for Imagery Hosting) MACHINE6 - "Federated" ArcGIS Server setup as the Imagery Hosting server

We have a couple deployments just like this deployment and have been running well.
One of the deployments started at v10.6.1 with just machine 1-4 (2 JSON files) and then was upgraded to v10.7.1 where machines 5-6 were added with 2 more JSON files. We have not attempted an upgrade on this system yet so unknown if the issue exists there

This system started at v10.7.1 with all 6 machines in the 4 JSON files. Install/license/config went just fine (at PSDSC v2.x) The upgrade seemed to run fine on JSON file 1,3,4 but the file 2 is the issue.

Here are the "ServerUpgrade--Error.txt":

12/16/2020 5:13:47 PM: PowerShell DSC resource ArcGIS_Install failed to execute Set-TargetResource functionality with error message: Failed to Install Server 12/16/2020 5:13:47 PM: The SendConfigurationApply function did not succeed.

Here are the "ServerUpgrade--Verbose.txt":

12/16/2020 4:55:00 PM: Perform operation 'Invoke CimMethod' with following parameters, ''methodName' = SendConfigurationApply,'className' = MSFT_DSCLocalConfigurationManager,'namespaceName' = root/Microsoft/Windows/DesiredStateConfiguration'. 12/16/2020 4:55:05 PM: An LCM method call arrived from computer with user sid . 12/16/2020 4:55:05 PM: []: LCM: [ Start Set ] 12/16/2020 4:55:05 PM: []: LCM: [ Start Resource ] [[ArcGIS_Install]ServerUpgrade] 12/16/2020 4:55:05 PM: []: LCM: [ Start Test ] [[ArcGIS_Install]ServerUpgrade] 12/16/2020 4:55:05 PM: []: [[ArcGIS_Install]ServerUpgrade] Installed Version 10.7.11595 12/16/2020 4:55:05 PM: []: LCM: [ End Test ] [[ArcGIS_Install]ServerUpgrade] in 0.4070 seconds. 12/16/2020 4:55:05 PM: []: LCM: [ Start Set ] [[ArcGIS_Install]ServerUpgrade] 12/16/2020 4:55:05 PM: []: [[ArcGIS_Install]ServerUpgrade] Installing Software using installer at <\server\share\file> 12/16/2020 5:13:47 PM: []: [[ArcGIS_Install]ServerUpgrade] Validating the Server Installation 12/16/2020 5:13:47 PM: []: [[ArcGIS_Install]ServerUpgrade] Installed Version 12/16/2020 5:13:47 PM: []: LCM: [ End Set ] [[ArcGIS_Install]ServerUpgrade] in 1124.5550 seconds. 12/16/2020 5:13:47 PM: []: LCM: [ End Set ] 12/16/2020 5:13:47 PM: Operation 'Invoke CimMethod' complete.

It almost appears the ArcGIS Server installation was corrupted and we are stuck trying to get it working.
There is only (1) .exe running as our service account called "ArcGISServer.exe".
The ArcGIS Server is not listening on port 6080 or 6443

We have tried a variety of things, without success so far...

1) We tried ... shutting down the v10.8.1 server overlay the original 10.7.1 configuration store Startup the v10.8.1 server attempt the PSDSC upgrade again RESULT - The server starts up in a bad state

2) We were going to try... Manual un-install of v10.8.1 Re-install v10.7.1 with config-store local Repoint the config-store to the UNC share Shutdown v10.7.1 Overlay the old v10.7.1 config store Start up the server Re-attempt the PSDSC execution upgrade

However.... it seems that the PSDSC 10.8.1 upgrade is somehow still trying to run. Every time we uninstall 10.8.1, it gets put back on after a reboot. I am no expert at PSDSC, but it appears that (at some point, possibly the reboot), the server does a PSDSC 'consistency check' and finds that the server is not in the state we wanted it in so it re-installs ArcGIS Server 10.8.1 but it starts up in the bad state described above.

So... At this point we are trying to figure out a way out of this situation. We know that we dont want to unfederate the server due to all the existing services and items being 'orphaned'.

Any guidance is appreciated. Thank you.

References

cameronkroeker commented 3 years ago

Hi @pfoppe,

I recommend checking the PowerShell DSC Location Configuration Manager (LCM) ConfigurationMode setting on all the target nodes (as well as the orchestrating node) in the deployment. By default I believe its set to 'ApplyAndMonitor', which means if the node drifts from the original desired state it may keep trying to bring it back. Therefore you may want to change the ConfigurationMode to 'ApplyOnly', so that it won't be constantly monitoring and trying to bring it back to the original desired state.

Here is a third party resource that may help: https://www.pluralsight.com/blog/it-ops/powershell-dsc-local-configuration-manager

And Microsoft's doc that explains more about the different ConfigurationMode settings: https://docs.microsoft.com/en-us/powershell/scripting/dsc/managing-nodes/metaconfig?view=powershell-7.1

During the upgrade perhaps the ArcGIS Server setup failed silently. May need to check the Windows Event Application logs to see if there are any error messages during the time it failed. One thing I would check is disk space to ensure there is enough in the install directory. If you're using the self extracting setup archive ensure there is enough space for the extraction as well (by default gets extracted to C:\Windows\Temp).

pfoppe commented 3 years ago

Hey @cameronkroeker ,

Thank you for the information. I did read up more on powershell and did start looking into this but I was able to fix it without changing any of the ConfigurationMode settings.

Ultimately, we did the following:

At that point, the ArcGIS Server still would not start up. If we completed a reboot, PSDSC would try to re-execute the 10.8.1 upgrade.

By digging through the underlying logs (ex - \\machine\C$\Program Files\ArcGIS\Server\framework\etc\service\logs) We found:

Tue Jan 12 11:53:52.630 MST 2021 Found flag file C:\Program Files\ArcGIS\Server\framework\etc\ArcGISServerUG.txt. Starting in upgrade mode.

We ended up deleting this file and the ArcGIS Server (at 10.7.1) would start up At that point we tried the PSDSC execution again and after a few attempts got it to upgrade correctly.

Everything else seems functional. I will try to dig further into some of those PSDSC commands that help highlight the current configurations/state as they seemed very informative... but more info to dig into than I currently have time for.

Thanks again... we can mark this as closed.