Timeout with webadaptor

simongis commented 1 year ago

Learning to deploy a base Enterprise 11.1 environment onto a single VM running Ubuntu 20.04 using Chef, and hitting an issue with setting up the portal web adaptor step.

Environment:

ArcGIS Enterprise 11.1 - Base Stack on single machine
Virtualbox for a homelab, making heavy use of snapshots to learn from mistakes
Fresh instance of Ubuntu 20.04 - original hostname wombat
Using a DNS Server, have assigned the machine FQDN: wombat.simongis.xyx
hostname command = wombat.simongis.xyz
dnsdomainname command = simongis.xyz
Have not changed any receipes, just the arcgis-enterprise-primary.json file

Issue: Everything in arcgis-enterprise-primary.json appears to run fine, until portal_wa recipe.

WARN: Util.wait_until_url_available timed out for https://wombat.simongis.xyz/portal/webadaptor after 1001.23 seconds. 
ERROR: Failed to configure Web Adaptor with Portal for ArcGIS. Expected process to exit with [0], but received '1'.

Can access and login to the portal via https://wombat.simongis.xyz:7443/arcgis/home/
Can ping wombat.simongis.xyz from another machine on the LAN
ERR_CONNECTION_REFUSED on https://wombat.simongis.xyz/portal/home
ERR_CONNECTION_REFUSED on https://wombat.simongis.xyz/portal/webadaptor

Can you provide any pointers on where I might be going wrong here?

End goal is to have a repeatable way of quickly deploying a fresh 11.1 Enterprise Environment on Linux for self-development, and share the steps to get there via a blog post (and things I learnt along the way). Eventua

Let me know if you prefer this type of issue raised on Esri Community

running_deployment_template.txt

arcgis-enterprise-primary.txt

portal service.log

Hosts file on Ubuntu

End goal is to have a repeatable way of quickly deploying a fresh 11.1 Enterprise Environment on Linux for self-development, and share the steps to get there via a blog post (and things I learnt along the way). Eventually get it working with a reverse proxy and fetching LetsEncrypt certs. That can come later.

cameronkroeker commented 1 year ago

Hi @simongis,

I wonder if the issue is related to not having a value specified for the tomcat attribute "keystore_type": "",. This should be set to PKCS12 ("keystore_type": "PKCS12",).

In the chef run logs I noticed:

 +    <Connector protocol="org.apache.coyote.http11.Http11Nio2Protocol"
    +               port="8443" maxThreads="200"
    +               scheme="https" secure="true" SSLEnabled="true"
    +               sslEnabledProtocols="TLSv1.3,TLSv1.2"
    +               keystoreFile="/opt/tomcat_arcgis_9.0.48/certificates/wombat.simon                                                                                        gis.xyz.pfx"
    +               keystorePass="wombat.simongis.xyz"
    +               keystoreType=""
    +               clientAuth="false" sslProtocol="TLS" />

Perhaps this is causing the tomcat SSL listner to not start properly. Let's check the following:

See if there are any clues in the tomcat logs catalina.out located in /opt/tomcat_arcgis/logs
Check to see if the following endpoints return a 200 status code (run from the node where the web adaptor is installed on):

curl http://wombat.simongis.xyz:8080/portal/webadaptor -v

curl http://wombat.simongis.xyz/portal/webadaptor -v

curl https://wombat.simongis.xyz:8443/portal/webadaptor --insecure -v

curl https://wombat.simongis.xyz/portal/webadaptor --insecure -v

Thanks, Cameron K.

simongis commented 1 year ago

Thanks for helping @cameronkroeker

This should be set to PKCS12 ("keystore_type": "PKCS12",).

Good spot. I am going for self signed for now and assumed this should be blank. Will get a fresh attempt going after running through your next steps to see if your theory hits the mark.

tomcat logs catalina.out

Looks like your spot on:

21-May-2023 06:12:31.301 SEVERE [main] org.apache.tomcat.util.net.SSLUtilBase.getStore Failed to load keystore type [] wi>
        java.security.KeyStoreException:  not found
                at java.base/java.security.KeyStore.getInstance(KeyStore.java:878)
                at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:184)
                at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.ja>
                at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:282)
                at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:97)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
                at org.apache.tomcat.util.net.Nio2Endpoint.bind(Nio2Endpoint.java:141)
                at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1208)
                at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1221)
                at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:603)
                at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:80)
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1046)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
                at org.apache.catalina.core.StandardService.initInternal(StandardService.java:561)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
21-May-2023 06:12:31.301 SEVERE [main] org.apache.tomcat.util.net.SSLUtilBase.getStore Failed to load keystore type [] wi>
        java.security.KeyStoreException:  not found

curl http://wombat.simongis.xyz:8080/portal/webadaptor -v curl http://wombat.simongis.xyz/portal/webadaptor -v Both 200. "statusText">This web adaptor is not configured with an ArcGIS Enterprise component

curl https://wombat.simongis.xyz:8443/portal/webadaptor --insecure -v Failed to connect to wombat.simongis.xyz port 8443: Connection refused

curl https://wombat.simongis.xyz/portal/webadaptor --insecure -v Failed to connect to wombat.simongis.xyz port 443: Connection refused

Let me restore snapshot, add the missing key and give it another whirl. Will report back and hopefully close issue! Thanks again. Do let me know if you prefer these fairly simple Qs being posted here or in a space in Esri Community.

cameronkroeker commented 1 year ago

Hi @simongis,

If you are going for a self-signed certificate then you can omit the attributes entirely from your json file, the cookbooks will then default to the values specified in /cookbooks/esri-tomcat/attributes/default.rb. For example:

https://github.com/Esri/arcgis-cookbook/blob/b6eba15c34cd5bbb192df830f395b3d5aea0e735/cookbooks/esri-tomcat/attributes/default.rb#L23

The attributes defined in your json file override these. So if you specify the attribute and leave the value blank it will override the default value with a blank value.

In my opinion submitting issues and questions here in GitHub is the preferred option.

Thanks, Cameron K.

simongis commented 1 year ago

Removing the lines worked. All services appear to be accessible and functioning as expected - thanks loads @cameronkroeker

Will now look at working out how to get some LetsEncrypt certs involved using something like Nginx Proxy Manager or go-acme - won't be bothering you about that side of things. Thanks!

Esri / arcgis-cookbook

Timeout with webadaptor #354