ArcGIS Enterprise HA: cannot access ArcGIS Server admin API on port 6443 because of redirection to WA

Biboba commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request

Module Version

3.1.1

Affected Resource(s)

ArcGIS_Server

Configuration Files

ArcGIS_Enterprise_HA.txt

Expected Behavior

Should be able to login to private ArcGIS Server Admin URL: https://mapsqa.company.com:6443/arcgis/admin/

Actual Behavior

When trying to login to private ArcGIS Server Admin URL: https://mapsqa.company.com:6443/arcgis/admin/ The page is returning a 302 page with redirection to WebAdaptor URL: https://mapsqa.company.com/arcgis/admin/login But as admin access has been disabled at web adaptor level, a 403 Forbidden is returned. So no login can be done.

I can successfully login on each VM though: https://mapsportalqa01.company.com:6443/arcgis/admin/ https://mapsportalqa02.company.com:6443/arcgis/admin/

Steps to Reproduce

Run the attached config. Once over, check the private ArcGIS Server Admin URL

Important Factoids

An HA proxy is configured to balance ArcGIS Server private port 6443 to each each: HAProxy: https://mapsqa.company.com:6443/arcgis balancing with X-forwarded-host to: VM1: https://mapsportalqa01.company.com:6443/arcgis VM2: https://mapsportalqa02.company.com:6443/arcgis

Rerunning, the DSC config does not solve the issue.

The only solution that works is to remove "WebContextURL" property from ArcGIS Server admin API: https://mapsportalqa01.company.com:6443/arcgis/admin/system/properties. Which is set to: https://mapsqa.company.com/arcgis

I don't know whether it is a bug from DSC or from ArcGIS Server itself because according to the documentation, it seems like this property should be set.

Also, whenever I run again the config, this property is applied once again so I need to update the property once again.

Checking at the output from DSC, the following URLs are summarized:

Portal Admin URL - https://mapsqa.company.com/geoportal/portaladmin
Server Admin URL - https://mapsportalqa01.company.com:6443/arcgis/admin
Server Manager URL - https://mapsqa.company.com/arcgis/manager
Server Rest URL - https://mapsqa.company.com/arcgis/rest

I wonder why the server Admin URL is not: https://mapsqa.company.com:6443/arcgis/admin

Also, accessing the manager private URL works: https://mapsqa.company.com:6443/arcgis/manager

Could it be related ?

Thanks !

cameronkroeker commented 3 years ago

Hello @Biboba,

I was not able to reproduce this issue. I tried the following combinations using HAProxy for my LB, PSDSC Module v3.1.1 and ArcGIS Enterprise 10.8.1:

Test 1

ServerContext = arcgis
AdminAccess = false
verified that the webcontexturl was set to https://haproxylb.domain.com/arcgis

Test 2

ServerContext = server
AdminAccess= false
verified that the webcontexturl was set to https://haproxylb.domain.com/server

In all scenarios the private URLs https://haproxylb.domain.com:6443/arcgis/admin, and https://haproxylb.domain.com:6443/arcgis/manager works.

I recommend checking the HAProxy configuration (haproxy.cfg). Normally, I wouldn't do this as it falls outside of the arcgis powershell module, but here is a snippet of what works for me:

frontend http-ex
        bind *:80
        bind *:443 ssl crt /etc/haproxy/wildcard.pem
        http-request set-header X-Forwarded-Host haproxylb.domain.com
        redirect scheme https code 301 if !{ ssl_fc }
        mode http
        use_backend portal_https if { path_beg /portal }
        use_backend server_https if { path_beg /arcgis }

frontend serverhttp-in
        bind *:6080
        bind *:6443 ssl crt /etc/haproxy/wildcard.pem
        acl network_allowed src ags-Node-01.domain.com ags-Node-02.domain.com
        tcp-request connection reject if !network_allowed
        http-request replace-value Host (.*):6080 \1:6443
        redirect scheme https code 301 if !{ ssl_fc }
        mode http
        default_backend server_in_https

backend server_https
        mode http
        http-request set-header X-Forwarded-Host haproxylb.domain.com
        acl hdr_location res.hdr(Location) -m found
        rspirep ^Location:\ https://ags-Node-01.domain.com/arcgis/(.*) Location:\ https://haproxylb.domain.com/arcgis/\1 if hdr_location
        rspirep ^Location:\ https://ags-Node-02.domain.com/arcgis/(.*) Location:\ https://haproxylb.domain.com/arcgis/\1 if hdr_location
        option httpchk GET /arcgis/rest/info/healthCheck
        #add one or more Server WA backend servers listening on port 443
        server ags-Node-01.domain.com:443 ags-Node-01.domain.com:443 check ssl verify none
        server ags-Node-02.domain.com:443 ags-Node-02.domain.com:443 check ssl verify none

backend server_in_https
        mode http
        acl hdr_location res.hdr(Location) -m found
        rspirep ^Location:\ https://ags-Node-01.domain.com:6443/arcgis/(.*) Location:\ https://haproxylb.domain.com:6443/arcgis/\1 if hdr_location
        rspirep ^Location:\ https://ags-Node-02.domain.com:6443/arcgis/(.*) Location:\ https://haproxylb.domain.com:6443/arcgis/\1 if hdr_location
        option httpchk GET /arcgis/rest/info/healthCheck
        #add one or more server backend servers listening on port 6443
        server ags-Node-01.domain.com:6443 ags-Node-01.domain.com:6443 check ssl verify none
        server ags-Node-02.domain.com:6443 ags-Node-02.domain.com:6443 check ssl verify none

Biboba commented 3 years ago

Many thanks @cameronkroeker ! I noticed that you do not set "X-Forwarded-Host" for 6443 while we are doing it in our case. In the documentation, it is stated:

When integrating your reverse proxy with ArcGIS Server or ArcGIS Web Adaptor, be aware that both of these components expect to see the following property set in the header sent by the reverse proxy server

Could that be the difference ? I will test asap and let you know the outcome but at first sight this is the main difference with our setup.

Thanks again for your investigation and for sharing your HAProxy config: much appreciated

cameronkroeker commented 3 years ago

Many thanks @cameronkroeker ! I noticed that you do not set "X-Forwarded-Host" for 6443 while we are doing it in our case. In the documentation, it is stated:
When integrating your reverse proxy with ArcGIS Server or ArcGIS Web Adaptor, be aware that both of these components expect to see the following property set in the header sent by the reverse proxy server
Could that be the difference ? I will test asap and let you know the outcome but at first sight this is the main difference with our setup.

Thanks again for your investigation and for sharing your HAProxy config: much appreciated

Hi @Biboba,

Yes, actually I think that very well could be the difference. I just added the X-forwarded-Host for 6443 and now am seeing the same redirect issue you originally reported. I think the documentation is alluding to the X-Forwarded-Host is needed when:

LB (443) -> WA 443 LB (443) -> AGS 6443

But X-Forwarded-Host is not needed when:

LB (6443) -> AGS 6443

Thanks, Cameron K.

Biboba commented 3 years ago

Hi @cameronkroeker,

Many thanks for these explanations ! I confirm that it works when removing X-Forwarded-Host header to 6443 :)

Thanks !

Biboba commented 3 years ago

On a side note, any idea why the following is logged:

Checking at the output from DSC, the following URLs are summarized:

Portal Admin URL - https://mapsqa.company.com/geoportal/portaladmin
Server Admin URL - https://mapsportalqa01.company.com:6443/arcgis/admin
Server Manager URL - https://mapsqa.company.com/arcgis/manager
Server Rest URL - https://mapsqa.company.com/arcgis/rest

I would expect "Server Admin URL" to be:

Server Admin URL - https://mapsqa.company.com:6443/arcgis/admin

Thanks !

cameronkroeker commented 3 years ago

On a side note, any idea why the following is logged:

Checking at the output from DSC, the following URLs are summarized:

Portal Admin URL - https://mapsqa.company.com/geoportal/portaladmin
Server Admin URL - https://mapsportalqa01.company.com:6443/arcgis/admin
Server Manager URL - https://mapsqa.company.com/arcgis/manager
Server Rest URL - https://mapsqa.company.com/arcgis/rest

I would expect "Server Admin URL" to be:

Server Admin URL - https://mapsqa.company.com:6443/arcgis/admin

Thanks !

This happened because we have "Server Admin URL" pointing to the Primary Server:

https://github.com/Esri/arcgis-powershell-dsc/blob/50a5cf49476a7059ea417dcc67f15c76e4d41522/Modules/ArcGIS/ArcGIS.psm1#L1478

https://github.com/Esri/arcgis-powershell-dsc/blob/50a5cf49476a7059ea417dcc67f15c76e4d41522/Modules/ArcGIS/ArcGIS.psm1#L1487

The current logic doesn't take account for when an ExternalLoadBalancer or InternalLoadBalancer is used. We will look to improve this logic in a future release, great catch!

Happy Automating, Cameron K.

Esri / arcgis-powershell-dsc