Base Deployment Multi-Machine Install Issue

nate-muller commented 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request

Module Version

4.1.0

Affected Resource(s)

ArcGIS_Server
ArcGIS_Portal
ArcGIS_DataStore
ArcGIS_FileStore

Configuration Files

{
    "AllNodes": [
        {
            "NodeName": "10.0.18.210",
            "DataStoreTypes": [
                "Relational"
            ],
            "Role": [
                "DataStore"
            ]
        },
        {
            "NodeName": "10.0.21.101",
            "DataStoreTypes": [
                "Relational"
            ],
            "Role": [
                "DataStore"
            ]
        },
        {
            "NodeName": "10.0.19.204",
            "Role": [
                "FileShare"
            ]
        },
        {
            "NodeName": "10.0.17.102",
            "Role": [
                "Server"
            ]
        },
        {
            "NodeName": "10.0.21.16",
            "Role": [
                "Server"
            ]
        },
        {
            "NodeName": "10.0.16.54",
            "Role": [
                "Portal"
            ]
        },
        {
            "NodeName": "10.0.20.120",
            "Role": [
                "Portal"
            ]
        },
        {
            "NodeName": "10.0.17.244",
            "Role": [
                "ServerWebAdaptor",
                "PortalWebAdaptor"
            ]
        },
        {
            "NodeName": "10.0.20.42",
            "Role": [
                "ServerWebAdaptor",
                "PortalWebAdaptor"
            ]
        }
    ],
    "ConfigData": {
        "Version": "10.9.1",
        "ServerContext": "server",
        "PortalContext": "portal",
        "FileShareLocalPath": "D:\\",
        "FileShareName": "\\\\FILE_SERVER\\y",
        "ServerRole": "GeneralPurposeServer",
        "LicenseManagerVersion": "2022.1",
        "DownloadPatches": true,
        "Credentials": {
            "ServiceAccount": {
                "Password": "<PASSWORD>",
                "UserName": "AGISE\\arcgis",
                "IsDomainAccount": true
            }
        },
        "Server": {
            "LicenseFilePath": "\\\\FILE_SERVER\\y\\License\\ArcGISGISServerAdvanced_ArcGISServer_1318811.prvc",
            "Installer": {
                "Path": "\\\\FILE_SERVER\\y\\Installers\\ArcGIS_Server_Windows_1091_180041.exe",
                "InstallDir": "C:\\Program Files\\ArcGIS\\Server",
                "InstallDirPython": "C:\\Python27"
            },
            "ExternalLoadBalancer": "<AWS_ALB_URL>",
            "EnableHTTPSOnly": true,
            "EnableHSTS": true,
            "ServerDirectoriesRootLocation": "\\\\FILE_SERVER\\y\\arcgisserver\\directories",
            "ConfigStoreLocation": "\\\\FILE_SERVER\\y\\arcgisserver\\config-store",
            "ServerLogsLocation": "D:\\arcgisserver\\logs",
            "PrimarySiteAdmin": {
                "UserName": "admin",
                "Password": "<PASSWORD>"
            }
        },
        "Portal": {
            "LicenseFilePath": "\\\\FILE_SERVER\\y\\License\\ArcGIS_Enterprise_Portal_1091_399769_20230425.json",
            "ExternalLoadBalancer": "<AWS_ALB_URL_2>",
            "PortalLicenseUserTypeId": "GISProfessionalAdvUT",
            "DisableAnonymousAccess": true,
            "EnableHSTS": true,
            "DefaultRoleForUser": "iAAAAAAAAAAAAAAA",
            "Installer": {
                "Path": "\\\\FILE_SERVER\\y\\Installers\\Portal_for_ArcGIS_Windows_1091_180052.exe",
                "InstallDir": "C:\\Program Files\\ArcGIS\\Portal",
                "ContentDir": "D:\\arcgisportal"
            },
            "ContentDirectoryLocation": "\\\\FILE_SERVER\\y\\arcgisportal\\content",
            "PortalAdministrator": {
                "UserName": "admin",
                "Email": "email@company.com",
                "Password": "<PASSWORD>",
                "SecurityQuestionIndex": 8,
                "SecurityAnswer": "vanilla"
            }
        },
        "DataStore": {
            "ContentDirectoryLocation": "D:\\arcgisdatastore",
            "EnableFailoverOnPrimaryStop": true,
            "EnablePointInTimeRecovery": true,
            "Installer": {
                "Path": "\\\\FILE_SERVER\\y\\Installers\\ArcGIS_DataStore_Windows_1091_180054.exe",
                "InstallDir": "C:\\Program Files\\ArcGIS\\DataStore",
                "InstallAllFeatures": false
            }
        },
        "WebAdaptor": {
            "AdminAccessEnabled": false,
            "Installer": {
                "Path": "\\\\FILE_SERVER\\y\\Installers\\ArcGIS_Web_Adaptor_Java_Linux_1091_180206.tar.gz"
            }
        }
    }
}

Expected Behavior

Howdy ESRI!

We are trying to run a multi-machine install for AGISE 10.9.1 in our AWS environment (running AGISE on EC2s) and we keep getting pretty well stonewalled. I expect that the DSC should run and install/license/configure the AGISE products specified in the JSON file above, but nothing happens and no ArcGIS products are installed on any machine, even after a restart.

We have done test run DSC runs with similar configurations in this environment before. We did a base deployment on a single machine (server/portal/webadapter) and that succeeded. We then uninstalled that using the -Mode Uninstall flag, and did a base deployment on two machines (server/portal/webadapter on the same machine as before, file store on another machine) and that succeeded.

Before we tried running the multi-machine script above, we have blown away and re-created all the EC2s that we did the prior DSC installs on.

Actual Behavior

Every time we run the DSC, we get absolutely nothing. All I've seen (for like weeks now, I'm starting to go a bit crazy) every time I try to run DSC is the following:

PS C:\Users\Administrator> Invoke-ArcGISConfiguration C:\script_files\BaseDeployment-MultiMachine.json -DebugSwitch
Dot Sourcing the Configuration:- ArcGISInstall
WARNING: It is not recommended to use domain credential for node '10.0.18.210'. In order to suppress the warning, you can add a property named 'PSDscAllowDomainUser' with a value of $true to your DSC
configuration data for node '10.0.18.210'.
WARNING: It is not recommended to use domain credential for node '10.0.21.101'. In order to suppress the warning, you can add a property named 'PSDscAllowDomainUser' with a value of $true to your DSC
configuration data for node '10.0.21.101'.
WARNING: It is not recommended to use domain credential for node '10.0.17.102'. In order to suppress the warning, you can add a property named 'PSDscAllowDomainUser' with a value of $true to your DSC
configuration data for node '10.0.17.102'.
WARNING: It is not recommended to use domain credential for node '10.0.21.16'. In order to suppress the warning, you can add a property named 'PSDscAllowDomainUser' with a value of $true to your DSC configuration
 data for node '10.0.21.16'.
WARNING: It is not recommended to use domain credential for node '10.0.16.54'. In order to suppress the warning, you can add a property named 'PSDscAllowDomainUser' with a value of $true to your DSC configuration
 data for node '10.0.16.54'.
WARNING: It is not recommended to use domain credential for node '10.0.20.120'. In order to suppress the warning, you can add a property named 'PSDscAllowDomainUser' with a value of $true to your DSC
configuration data for node '10.0.20.120'.
Starting DSC Job:- ArcGISInstall
VERBOSE: Time taken for configuration job to complete is 0.234 seconds
NATE NATE NATE DSC Job:
Logs Directory: C:\Users\Administrator\Logs
Finished DSC Job:- ArcGISInstall. Time Taken - 00:00:00.2735514
ArcGISInstall -
PS C:\Users\Administrator>

That's it! As you can see, I've even taken to adding my own Write-Information statements to ArcGIS.psm1 to try and get more information, but those aren't doing me much good. It appears a DSC job isn't even starting, as the Write-Information statement I added above is (ArcGIS.psm1, added this statement after line 259):

Write-Information -InformationAction Continue "NATE NATE NATE DSC Job: $Job"

I have taken the following prerequisite steps:

Set the Local ConfigurationMode to ApplyOnly on the orchestrating node
Ran the Set-ExecutionPolicy RemoteSigned and winrm quickconfig commands on all nodes
Ran the winrm s winrm/config/client '@{TrustedHosts="<IP address goes here>"}' command on the orchestrating node, with all the IPs listed above in the trusted hosts list.
Cleared the DSC cache and restarted the machine several times between runs
Removed the SpatioTemporal BDS and License Manager from the config file above (it still didn't work after I removed them)
Set the Local Configuration Manager ActionAfterReboot to StopConfiguration on the orchestrating node.
Set the Local Configuration Manager DebugMode to ForceModuleImport, All, and $true at various points in time on the orchestrating node.
Read basically every issue I could find in here that I thought was remotely related with no luck
I can get to the file share from the orchestrating node, I tested from one of the server nodes and was successful as well. All nodes are part of the domain and have the same security group allowing file share access over SMB port 445 so I assume all named nodes in the config above can get to the file share.

Please help me ESRI, you're my only hope! Before I drive myself and our ESRI technical team crazy!

Important Factoids

Running on AWS EC2s
All Nodes are domain joined to a Managed AWS Windows AD Domain called AGISE
All computer accounts have r/w permissions on \\FILESHARE\y
The AGISE\arcgis account has r/w permissions on \\FILESHARE\y

References

None

cameronkroeker commented 1 year ago

Hi @nate-muller,

You may need to pass in the -Credential flag with the Invoke-ArcGISConfiguration command when NodeName is using IP addresses instead of hostnames.

https://github.com/Esri/arcgis-powershell-dsc/wiki/Using-Internet-Protocol-(IP)-address-instead-of-Fully-Qualified-Domain-Names-(FQDN)#install-your-deployment

A couple of other observations (unlikely not related to the issue you describe but these will cause failures/issues later down the line).

ConfigData.FileShareName should be something like this:
```
"FileShareName": "y" 
```
The ArcGIS Module does not support installing the Java WA:

"Path": "\\\\FILE_SERVER\\y\\Installers\\ArcGIS_Web_Adaptor_Java_Linux_1091_180206.tar.gz"

Ensure the ConfigData.ExternalLoadBalancer value is set to the FQDN and not the actual URL. For example:

"ExternalLoadBalancer": "lbAlias.domain.com"

Thanks, Cameron K.

nate-muller commented 1 year ago

Thanks for getting back to me during UC Cameron. I am trying your Credential suggestion now!

nate-muller commented 1 year ago

Cameron,

Your hints/suggestions worked great! The following ended up being my issues:

I did indeed need to include the -Credential parameter. I ended up passing the fully qualified username of the domain admin (e.g. AGISE\admin) as the the value of the parameter, and then entered the password when prompted.
I had to open the WinRM ports (5985 and 5986) on all machines that I am installing stuff onto via DSC.
I validated that the DSC powershell module was installed on all servers. Since I am targeting windows server 2022, and DSC is included by default on win server 2022, I ended up not needing to take any action here.

Thanks so much, I will close the ticket!

Esri / arcgis-powershell-dsc