search

Esri / arcgis-cookbook

Chef cookbooks for ArcGIS
Apache License 2.0
300 stars 116 forks source link

Unable to Authorize Portal on Linux EC2 using Cinc #302

Closed DGrady83 closed 2 years ago

DGrady83 commented 2 years ago

We are attempting to install ArcGIS Enterprise on a Linux EC2 machine and after working through a few issues, we are now stuck on the Portal Authorization step.

Below is the error getting thrown.

arcgis_enterprise_portal[Authorize Portal for ArcGIS] action authorize[2022-03-02T10:04:55-05:00] WARN: Util.wait_until_url_available timed out for https://[maskedIPaddress]:7443/arcgis/portaladmin after 2206.21 seconds.

================================================================================
Error executing action `authorize` on resource 'arcgis_enterprise_portal[Authorize Portal for ArcGIS]'
================================================================================

Net::HTTPFatalError
-------------------
503 "Service Unavailable"

Cookbook Trace: (most recent call first)
----------------------------------------
/root/.cinc/local-mode-cache/cache/cookbooks/arcgis-enterprise/libraries/portal_admin_client.rb:854:in `send_request'
/root/.cinc/local-mode-cache/cache/cookbooks/arcgis-enterprise/libraries/portal_admin_client.rb:750:in `is_user_type_licensing'
/root/.cinc/local-mode-cache/cache/cookbooks/arcgis-enterprise/providers/portal.rb:256:in `block in class_from_file'

Resource Declaration:
---------------------
# In /root/.cinc/local-mode-cache/cache/cookbooks/arcgis-enterprise/recipes/portal.rb

 54: arcgis_enterprise_portal 'Authorize Portal for ArcGIS' do
 55:   authorization_file node['arcgis']['portal']['authorization_file']
 56:   authorization_file_version node['arcgis']['portal']['authorization_file_version']
 57:   user_license_type_id node['arcgis']['portal']['user_license_type_id']
 58:   portal_url node['arcgis']['portal']['url']
 59:   username node['arcgis']['portal']['admin_username']
 60:   password node['arcgis']['portal']['admin_password']
 61:   action :authorize
 62: end
 63:
Compiled Resource:
------------------
# Declared in /root/.cinc/local-mode-cache/cache/cookbooks/arcgis-enterprise/recipes/portal.rb:54:in `from_file'

arcgis_enterprise_portal("Authorize Portal for ArcGIS") do
  action [:authorize]
  default_guard_interpreter :default
  declared_type :arcgis_enterprise_portal
  cookbook_name "arcgis-enterprise"
  recipe_name "portal"
  setup_options ""
  authorization_file "/opt/software/authorization_files/10.9.1/ArcGIS_Enterprise_Portal_109_353098_20211203.json"
  authorization_file_version "10.9.1"
  user_license_type_id ""
  portal_url "https://[maskedIPaddress]:7443/arcgis"
  username "portaladmin"
  password "*********"
  content_store_type "fileStore"
  content_store_provider "FileSystem"
  log_level "WARNING"
  system_properties {}
  user_store_config {}
  role_store_config {}
end

System Info:
------------
chef_version=17.9.52
platform=redhat
platform_version=8.5
ruby=ruby 3.0.3p157 (2021-11-24 revision 3fb7d2cadc) [x86_64-linux]
program_name=/bin/cinc-client
executable=/opt/cinc/bin/cinc-client
cameronkroeker commented 2 years ago

Hello @DGrady83,

Based on the error the following url is timing out or not accessible https://[maskedIPaddress]:7443/arcgis/portaladmin. Let's confirm that portal service is started, and the url is accessible. Some troubleshooting tips:

This command will check the status of the portal service:

sudo systemctl status arcgisportal.service

If portal is stopped, try to start it using:

sudo systemctl start arcgisportal.service

To stop portal:

sudo systemctl stop arcgisportal.service

To restart portal:

sudo systemctl restart arcgisportal.service

If you don't have a local browser, can try running a curl command to check if the portal url is accessible:

curl https://[maskedIPaddress]:7443/arcgis/portaladmin?f=json --insecure -v

or

curl https://[maskedIPaddress]:7443/arcgis/portaladmin/healthCheck?f=json--insecure -v

You may also want to check the portal logs to see if there are any startup errors, check to see if portal processes are running (ps -ef | grep portal or ps -ef | grep java), check to see if there are any processes listening on port 7443 (netstat -anpe | grep "7443" | grep "LISTEN").

To confirm this is running on RedHat Enterprise Linux 8.5? And cookbooks v3.8.0 with ArcGIS Enterprise 10.9.1?

Thanks, Cameron K.

DGrady83 commented 2 years ago

Hi Cameron - you are correct. Running RHEL 8.5 w/ Cookbooks 3.8 and ArcGIS 10.9.1

Portal service appears to be in a running state:

[root@ip-** cinc]# sudo systemctl status arcgisportal.service ● arcgisportal.service - Portal for ArcGIS Service Loaded: loaded (/etc/systemd/system/arcgisportal.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2022-03-02 13:37:26 EST; 1h 16min ago Process: 1295 ExecStart=/opt/arcgis/portal/startportal.sh (code=exited, status=0/SUCCESS) Tasks: 139 (limit: 204120) Memory: 1.2G CGroup: /system.slice/arcgisportal.service ├─1369 /opt/arcgis/portal/framework/runtime/jre//bin/java -DARCGIS_PORTAL_NODEAGENT=true -classpath /opt/arcgis/portal/framework/lib/arcgis-nodeagent.jar:/opt/arcgis/portal/framework/lib/commons-configuration-1.9.jar:/opt/arc> └─2237 /opt/arcgis/portal/framework/runtime/jre/bin/java -Djava.util.logging.config.file=/opt/arcgis/portal/framework/runtime/tomcat/conf/logging.properties -Dportal=true -Dprofile=portal -Dspring.profiles.active=indexserver >

Mar 02 13:37:21 ip-**.ec2.internal systemd[1]: Starting Portal for ArcGIS Service... Mar 02 13:37:26 ip-**.ec2.internal systemd[1]: Started Portal for ArcGIS Service.

I do see the 503 error when I try accessing the below command though (note the asterisks are in place of the machine's IP)

[root@ip-** cinc]# curl https://**********:7443/arcgis/portaladmin?f=json --insecure -v

cameronkroeker commented 2 years ago

Thanks @DGrady83. Looks like there is some sort of proxy server the request is going through, perhaps a forward proxy? May need to add an exception to the proxy to allow the connection.

DGrady83 commented 2 years ago

Hi Cameron - I ended up adding the machine's IP to the no_proxy setting and it seemed like that is what ultimately got me past the Authorization issue. However, I am now getting an error when trying to configure WA

Recipe: arcgis-enterprise::install_portal_wa

Recipe: arcgis-enterprise::portal_wa

cameronkroeker commented 2 years ago

Hi @DGrady83, glad to hear adding the machine's IP to the no_proxy setting allowed portal to get past the authorization issue and site created.

The new error related to the portal web adaptor registration could be caused by a few different reasons. Looks like the request to https://[maskedIP]/portal/webadaptor is timing out so I suspect issue is related to either the tomcat web server and/or firewall\networking.

Check if https://[maskedIP]/portal/webadaptor can be reached (I suspect this will give a timeout error):

curl https://[maskedIP]/portal/webadaptor --insecure -v

Check if https://[maskedIP]:8443/portal/webadaptor can be reached:

curl https://[maskedIP]:8443/portal/webadaptor --insecure -v

If neither of these url's can be reached check to ensure tomcat service is started and running:

Check tomcat logs for any failures, default location is: /opt/tomcat_arcgis/logs

Check to ensure the correct\valid certificate is being used by tomcat in the server.xml ssl connector (if its ssl related though should see related errors in the tomcat logs):

/opt/tomcat_arcgis/conf/server.xml

Here is an example:

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
               port="8443" maxThreads="200"
               scheme="https" secure="true" SSLEnabled="true"
               sslEnabledProtocols="TLSv1.3,TLSv1.2"
               keystoreFile="/certificates/wild-card.pfx"
               keystorePass="certpassword"
               keystoreType="pkcs12"
               clientAuth="false" sslProtocol="TLS" />

If 8443 can be reached then lets check firewalld settings. By default the esri-tomcat cookbook will configure firewalld port forwarding (443->8443 and 80->8080) for RHEL. If firewalld service is running/enabled prior to running chef then it does not open any ports, as we assume that this has been pre-configured by the user and it will only add the port forwarding rules.

However, if firewalld isn't installed or started prior to running chef then we will install, start, open all tcp ports, and add the port forwarding rules:

https://github.com/Esri/arcgis-cookbook/blob/1c7f39341a4a967c6c73a20ee0b6bbb5b2a3dc53/cookbooks/esri-tomcat/recipes/firewalld.rb#L46-L50

Let's ensure the required ports are open in firewalld, and port forwarding rules are set.

Status of firewalld service

systemctl status firewalld.service

List all zones and ports/services that are open

firewall-cmd --list-all

Hope this helps!

Thanks, Cameron K.

DGrady83 commented 2 years ago

OK we are getting closer. I think there was an issue with my SSL certificate so I created a new one and placed it on the server and am now past the WA error. However, I am now getting blocked when the installation is trying to authorize Server. I do have a valid prvc file in the location specified in my arcgis-enterprise-primary.json file I am running ("authorization_file": "/opt/software/authorization_files/10.9.1/ArcGISGISServerAdvanced_StagingArcGISServer_1149435.prvc", )

Recipe: arcgis-enterprise::server
  * arcgis_enterprise_server[Start ArcGIS Server after upgrade] action start
    * service[arcgisserver] action enable (up to date)
    * service[arcgisserver] action start (up to date)

  * arcgis_enterprise_server[Authorize ArcGIS Server] action authorize[2022-03-04T13:39:24-05:00] ERROR: --------------------------------------------------------------------------
Starting the ArcGIS Software Authorization Wizard

Run this script with -h for additional information.
--------------------------------------------------------------------------
Not Authorized.

[2022-03-04T13:39:24-05:00] ERROR: ArcGIS Server is not authorized. Retrying software authorization...
[2022-03-04T13:44:38-05:00] ERROR: --------------------------------------------------------------------------
Starting the ArcGIS Software Authorization Wizard

Run this script with -h for additional information.
--------------------------------------------------------------------------
Not Authorized.

[2022-03-04T13:44:38-05:00] ERROR: ArcGIS Server is not authorized. Retrying software authorization...
[2022-03-04T13:47:53-05:00] ERROR: --------------------------------------------------------------------------
Starting the ArcGIS Software Authorization Wizard

Run this script with -h for additional information.
--------------------------------------------------------------------------
Not Authorized.
cameronkroeker commented 2 years ago

Hi @DGrady83,

I suspect either the ArcGIS Server Authorization Wizard (SAW) is not able to establish a connection with the Esri authorization server, perhaps the proxy that is sitting in front is blocking the request, or there is something not right with the authorization file (perhaps permissions, or the file is corrupt).

One thing we can do is try to manually run the command that chef is using to see if there is a more useful error that is displayed. As the arcgis user (sudo su arcgis) try running the following command:

/opt/arcgis/server/tools/authorizeSoftware -f /opt/software/authorization_files/10.9.1/ArcGISGISServerAdvanced_StagingArcGISServer_1149435.prvc

Then run this command to check the status:

/opt/arcgis/server/tools/authorizeSoftware -s

If it turns out the request is being blocked, and an exception can't be made, we could try the offline authorization method.

Thanks, Cameron K.

DGrady83 commented 2 years ago

Hi Cameron - thank you for the reply. Here is where I am at now

[arcgis@ip-[MaskedIP] cloud-user]$ /opt/arcgis/server/tools/authorizeSoftware -f /opt/software/authorization_files/10.9.1/ArcGISGISServerAdvanced_StagingArcGISServer_1149435.prvc
--------------------------------------------------------------------------
Starting the ArcGIS Software Authorization Wizard

Run this script with -h for additional information.
--------------------------------------------------------------------------
Unable to load library icui18n "Cannot load library icui18n: (icui18n: cannot open shared object file: No such file or directory)"

Connecting to Esri...
Sending Authorization Information...
Receiving Authorization...
Receiving Authorization Failed!
Error authorizing with the following file

/opt/software/authorization_files/10.9.1/ArcGISGISServerAdvanced_StagingArcGISServer_1149435.prvc
DGrady83 commented 2 years ago

Note that I do see icui18n listed in these folders:

[arcgis@ip-[MaskedIP] cloud-user]$ locate icui18n /opt/arcgis/portal/framework/runtime/pgsql/lib/libicui18n.so /opt/arcgis/portal/framework/runtime/pgsql/lib/libicui18n.so.58 /opt/arcgis/portal/framework/runtime/pgsql/lib/libicui18n.so.58.3 /opt/arcgis/portal/usr/arcgisportal/pgsql_12.4/lib/libicui18n.so /opt/arcgis/portal/usr/arcgisportal/pgsql_12.4/lib/libicui18n.so.58 /opt/arcgis/portal/usr/arcgisportal/pgsql_12.4/lib/libicui18n.so.58.3 /usr/lib64/libicui18n.so.60 /usr/lib64/libicui18n.so.60.3

cameronkroeker commented 2 years ago

Hi @DGrady83,

Authorizing ArcGIS Server with a *.prvc file requires an internet connection. I suspect that the outbound request to the Esri Authorization server (service.esri.com) is being blocked based on the error: "Receiving Authorization Failed".

Option A: Make a temporary proxy server/firewall exception to allow internet access Try making an exception for service.esri.com, or allow all internet access until the authorization is complete.

or

Option B: Perform offline authorization method

https://enterprise.arcgis.com/en/server/latest/install/linux/silently-install-arcgis-server.htm#ESRI_SECTION2_F12CF18F11F444AB90FD53CF86A054F2

  1. Generate an authorization file to be uploaded to the Esri website or email. 
    /opt/arcgis/server/tools/authorizeSoftware -f /opt/software/authorization_files/10.9.1/ArcGISGISServerAdvanced_StagingArcGISServer_1149435.prvc -e EMAIL_ADDRESS -o AUTHORIZATION_FILE_OUTPUT
  2. Once completed, update the attribute arcgis.server.authorization_file in your chef json file to point to the received .ecp authorization file to complete the authorization process.
"authorization_file": "/opt/software/authorization_files/10.9.1/received_ecp_file.ecp"

Thanks, Cameron K.

DGrady83 commented 2 years ago

Hi Cameron - what are the steps for Option A? I'm fairly new to Linux so not sure how to do that.

cameronkroeker commented 2 years ago

Hi Cameron - what are the steps for Option A? I'm fairly new to Linux so not sure how to do that.

Hi @DGrady83,

Unfortunately I am not sure either as this is a bit out of my wheel house as well.

Do you have an IT or Networking department within your organization you can consult with? If not then Option B might be the best route.

Regards, Cameron K.

DGrady83 commented 2 years ago

Hi @cameronkroeker - I actually tried to perform the offline authorization but even when I do that I still get this error

Starting the ArcGIS Software Authorization Wizard

Run this script with -h for additional information.
--------------------------------------------------------------------------
Unable to load library icui18n "Cannot load library icui18n: (icui18n: cannot open shared object file: No such file or directory)"
cameronkroeker commented 2 years ago

Hi @cameronkroeker - I actually tried to perform the offline authorization but even when I do that I still get this error

Starting the ArcGIS Software Authorization Wizard

Run this script with -h for additional information.
--------------------------------------------------------------------------
Unable to load library icui18n "Cannot load library icui18n: (icui18n: cannot open shared object file: No such file or directory)"

Interesting! Did it at least create the output file? Let's see if there are any additional clues or hints in the following log file:

opt/arcgis/server/.Setup/softwareauthorization.log

DGrady83 commented 2 years ago

Unfortunately it did not even create the output file. Here is the log file

[root@ip-[maskedIP .Setup]# more softwareauthorization.log Unable to load library icui18n "Cannot load library icui18n: (icui18n: cannot open shared object file: No such file or directory)"

Connecting to Esri... Sending Authorization Information... Receiving Authorization... Receiving Authorization Failed! Error authorizing with the following file

/opt/software/authorization_files/10.9.1/ArcGISGISServerAdvanced_StagingArcGISServer_1149435.prvc

cameronkroeker commented 2 years ago

Thanks @DGrady83. It shouldn't be reaching out to the authorization server with the offline method, so I am wondering if those log messages are from the previous attempts.

To confirm you ran the following command as the arcgis user (sudo su arcgis), but replaced EMAIL_ADDRESS with a valid email, and AUTHORIZATION_FILE_OUTPUT with your desired output file name and path for it to be saved to? Ensure the path for the output file is one that the arcgis account has access to, like /home/arcgis.

For Example:

/opt/arcgis/server/tools/authorizeSoftware -f /opt/software/authorization_files/10.9.1/ArcGISGISServerAdvanced_StagingArcGISServer_1149435.prvc -e your-email@domain.com -o /home/arcgis/your-output-file-name

Thanks, Cameron K.

DGrady83 commented 2 years ago

Hi Cameron - I did the offline method (not preferred) and it seemed to work. I re-ran the installer as root with the .json file btw. However, I am now seeing a permission error but it doesn't specify what folder it needs to write to. Shouldn't the installer automatically adjust permissions accordingly?

DGrady83 commented 2 years ago

And this is the log file now.

[root@ip-[MaskedIP] .Setup]# more softwareauthorization.log Unable to load library icui18n "Cannot load library icui18n: (icui18n: cannot open shared object file: No such file or directory)" The following features were authorized

svradv svrenterprise networkserver arcsdeserver svradv_4

DGrady83 commented 2 years ago

2022-03-08_13-13-13

@cameronkroeker - so I am past the authorization error but now I keep getting these timeout warnings during the Federation process. Any idea what might be going on?

cameronkroeker commented 2 years ago

@DGrady83,

If the directory specified for arcgis.server.directories_root attribute is a local path, the cookbooks will create it and assign the arcgis user 700 permissions:

https://github.com/Esri/arcgis-cookbook/blob/1c7f39341a4a967c6c73a20ee0b6bbb5b2a3dc53/cookbooks/arcgis-enterprise/recipes/server.rb#L90-L99

From the log messages it looks like it did create and assign the permissions:

directory[/gisdata/arcgisserver] action create

change mode from '0755' to '0700'
change owner from 'root' to 'arcgis'
restore selinux security context

However, the "restore selinux security context" message is interesting to me though. Is selinux set to enforcing? If it is perhaps this is impacting the site creation.

Thanks, Cameron K.

cameronkroeker commented 2 years ago

2022-03-08_13-13-13

@cameronkroeker - so I am past the authorization error but now I keep getting these timeout warnings during the Federation process. Any idea what might be going on?

Looks like https://<blurred out>.com/server/rest/healthcheck?f=json is timing out. This is the server web adaptor url, is it reachable?

It is also timing out on https://<blurred out>.com:6443/arcgis/admin?f=json. This is the arcgis site url, is it reachable?

Thanks, Cameron K.

DGrady83 commented 2 years ago

Not able to reach the URL on a browser either. Get the following message in Chrome

2022-03-08_16-06-04

DGrady83 commented 2 years ago

@cameronkroeker - do we need to manually import the certificate (i.e. using keystore) or does the installer take care of that?

cameronkroeker commented 2 years ago

@cameronkroeker - do we need to manually import the certificate (i.e. using keystore) or does the installer take care of that?

Hi @DGrady83,

If you define the following attributes in you chef config json file, the cookbooks will import them automatically:

Portal node['arcgis']['portal']['keystore_file'] = Path to PKSC12 keystore file (.pfx) with SSL certificate for Portal for ArcGIS. Default value is nil. node['arcgis']['portal']['keystore_password'] = Keystore file password for Portal for ArcGIS. Default value is nil. node['arcgis']['portal']['cert_alias'] = SSL certificate alias for Portal for ArcGIS. Default alias is composed of these values:node['arcgis']['portal']['domain_name']. node['arcgis']['portal']['root_cert'] = Portal for ArcGIS root certificate. Default value is ''. node['arcgis']['portal']['root_cert_alias'] = Portal for ArcGIS root certificate alias. Default value is ''.

ArcGIS Server

node['arcgis']['server']['keystore_file'] = Path to PKSC12 keystore file (.pfx) with SSL certificate for ArcGIS Server. Default value is nil. node['arcgis']['server']['keystore_password'] = Keystore file password for ArcGIS Server. Default value is nil. node['arcgis']['server']['cert_alias'] = SSL certificate alias for ArcGIS Server. Default alias is composed of these values: node['arcgis']['server']['domain_name']. node['arcgis']['server']['root_cert'] = ArcGIS Server root CA certificate file path. Default value is ''. node['arcgis']['server']['root_cert_alias'] = ArcGIS Server root CA certificate alias. Default value is ''.

Tomcat

node['tomcat']['keystore_file'] = Optional: Path to the keystore file. If not provided, a new file and a self-signed certificate will be created. node['tomcat']['keystore_password'] = Optional: Password to the keystore.

Thanks, Cameron K.

cameronkroeker commented 2 years ago

Not able to reach the URL on a browser either. Get the following message in Chrome

2022-03-08_16-06-04

This is a bit tricky/complex to troubleshoot via GitHub. If you open an Esri Technical Support case perhaps we can take a deeper dive via a screen share/call.

cameronkroeker commented 2 years ago

closing issue, as we worked with Esri Support to resolve all related issues.

Ayushyadav19 commented 2 years ago

Hi @DGrady83,

We are deploying the Base ArcGIS Enterprise environment on Linux RHEL 8 on AWS ec2 instance and when we are running the chef script it throws the below error after installing the portal : image

I believe that this is something related to the Linux RHEL subscription. As you installed the ArcGIS Enterprise environment on the Linux amazon ec2 instance I just wanted to know if is there any subscription or license required for the Linux ec2 instance before deploying the ArcGIS enterprise environment.

Thanks Ayush

DGrady83 commented 2 years ago

Hi @Ayushyadav19 - Can't say that I'm familiar with that particular error unfortunately. Our EC2's are all managed by our organizations AWS account.

DGrady83 commented 1 year ago

Hi Cameron – we are looking at deploying for ArcGIS on multiple machines (high availability). Is that not an option using Cookbooks 4.0? I can only find it for 3.7

Thank you Dan

cameronkroeker commented 1 year ago

Hi Cameron – we are looking at deploying for ArcGIS on multiple machines (high availability). Is that not an option using Cookbooks 4.0? I can only find it for 3.7 Thank you Dan

Hi @DGrady83,

Yes, it's possible with v4.0.0 as well:

https://esri.github.io/arcgis-cookbook/templates/arcgis-enterprise-base/11.0.html

Thanks, Cameron K.