connection to internal gateway not working due to connection-type=notunnel

[kvernNC]

Hi, if it's not the right place to ask this, let me know.

We use 2 PAN on our network: one for all external access, and one for all internal access. GlobalProtect Portal is configured only on the external, and each as a Gateway configured.

When I use openconnect on an external network, I am able to connect, no problem, it works well.

The mean of an internal gateway, is to populate user-id information into the palo alto; We use it as an user-id agent deployed on all users computers. This help us manage internal access per user and groups in the firewalls rules instead of IP. There are others way to populate userid information, but we found this one to be the more accurate. Only issue for us as the moment is our few linux clients are not able to connect to the internal gateway.

Configuration of mixed external and internal gateway is documented on Palo Alto website.

Connection to internal gateway with official windows client works great; client is dynamically choosing the gateway it needs according to a dns request and the client IP.

When connecting using openconnect, my internal gateway is not available as a choice of gateway servers available but It is described in the XML file when using --dump.

offering:

...
POST https://mycorporate.com/global-protect/getconfig.esp
1 gateway servers available:
  external-GW (mycorporate.com)
Please select GlobalProtect gateway.
...

XML:

    <internal-host-detection>
        <ip-address>10.y.y.y</ip-address>
        <host>internal-server.int</host>
    </internal-host-detection>
    <gateways>
        <cutoff-time>5</cutoff-time>
        <internal>
            <list>
                <entry name="10.x.x.x">
                    <description>internal-GW</description>
                    <source-ip-v4>
                        <member>10.0.0.1-10.255.255.254</member>
                        <member>172.16.0.1-172.31.255.254</member>
                        <member>192.168.0.1-192.168.255.254</member>
                    </source-ip-v4>
                </entry>
            </list>
        </internal>
        <external>
            <list>
                <entry name="mycorporate.com">
                    <priority-rule>
                        <entry name="Any">
                            <priority>1</priority>
                        </entry>
                    </priority-rule>
                    <priority>1</priority>
                    <description>external-GW</description>
                </entry>
            </list>
        </external>
    </gateways>

Is this an expected behavior to ignore internal server or I'm having an issue with my configuration ?

I forced the connection on the internal gateway, authentication is working but then openconnect stop:

...
Connected to HTTPS on 10.x.x.x

: @username
: @password
POST https://10.x.x.x/ssl-vpn/login.esp
GlobalProtect login returned authentication-source=(empty_authprofile)
GlobalProtect login returned connection-type=notunnel (expected tunnel)
Failed to parse server response
Failed to obtain WebVPN cookie

For my use case, notunnel is expected, as we use globalprotect only as an userid agent, and we don't want to etablish a tunnel because all network resources are already available via routing.

On the internal firewall, as authentication was successful, user-id is correctly informed of my username/ip address in his database, but it will keep it until a timeout is reached (defaut is 45min).

Is this possible to allow connection-type=notunnel, and keeping the ssl session opened to have a sort of keepalive ?

Let me know if you need logs or testing a patch.

Regards,

[dlenski]

Thanks for the thorough and useful description. (By the way, since GP support has been merged into the official releases as of v8.00, might want to move this over to https://gitlab.com/openconnect/openconnect/issues at some point…)

Is this an expected behavior to ignore internal server or I'm having an issue with my configuration ?

I had never seen the possibility of an <internal> gateway before, nor had I ever seen connection-type=notunnel, so openconnect doesn't look for these possibilities. :man_shrugging:

For my use case, notunnel is expected, as we use globalprotect only as an userid agent, and we don't want to etablish a tunnel because all network resources are already available via routing.

Is this possible to allow connection-type=notunnel, and keeping the ssl session opened to have a sort of keepalive ?

Hmmm… let me make sure I understand here. What you want openconnect to do when it connects via an internal gateway with connection-type=notunnel is:

1. :heavy_check_mark: Do authenticate to the gateway as usual (via /ssl-vpn/login.esp)
2. :heavy_check_mark: Do notice that the gateway sends connection-type=notunnel and behave differently.
3. Then what…?
   • What about any configuration info in /ssl-vpn/getconfig.esp? Does the official client pull that and use its contents?
   • Apparently the (already-authenticated) HTTPS/TLS session needs to be kept alive in order to achieve the useful purpose of the "tunnel-less" connection, which is to prove the association between the IP address and a specific user. But how should that HTTPS/TLS session be kept alive? How does the official client keep it alive with no meaningful traffic to send over it? This is not apparent to me.

Let me know if you need logs or testing a patch.

Do you know how to run mitmproxy or something similar to get more details on what the official client is doing? Might also be possible to get useful information from the Windows client by turning up the PanGPS service logging to the highest level, and poring through the (very verbose) PanGPS.log…

[kvernNC]

Thanks for the encouraging answer.

your understanding is correct.

1. Do authenticate to the gateway as usual (via /ssl-vpn/login.esp)

already working.

2. Do notice that the gateway sends connection-type=notunnel and behave differently.

Exactly.

3. Then what…?

I was hoping you would have a clue on this. Maybe continually sending hip report is enough to update the timeout on the firewall.

Here is an anomyzed PanGPS.log of my windows VM after rebooting. I'm not finding getconfig content in it, but it may help you understand better. link

As a network administrator and a daily linux user, I should be able to run MITMproxy and capture all the xml that are exchanged between client and firewall. It will take someday for me, as I don't want to messing with my workstation environment and I can access to internal gateway only when at the office, obviously.

I feel thrilled every time I can help an opensource project, i'll do my best if you have the time to implement this enhancement.

[dlenski]

Thanks, that's helpful.

3. Then what…?

I was hoping you would have a clue on this. Maybe continually sending hip report is enough to update the timeout on the firewall.

:joy: This may dovetail nicely with https://gitlab.com/openconnect/openconnect/merge_requests/56, actually.

Does your portal's /global-protect/getconfig.esp response include a HIP collection interval? Something like this…

    <hip-collection>
        <hip-report-interval>3600</hip-report-interval>
        <max-wait-time>60</max-wait-time>
        <collect-hip-data>yes</collect-hip-data>
                ...
    </hip-collection>

[kvernNC]

I've been able to capture traffic between my workstation and portal gateway with wireshark. As I am one of the palo alto administrator, I've got it certificate private key, so I was able to decypher it without MITMproxy.

here is the xml response from the portal: https://pastebin.com/Q7cuD25D

Yes, I have the same hip-collection :

    <hip-collection>
        <hip-report-interval>3600</hip-report-interval>
        <max-wait-time>20</max-wait-time>
        <collect-hip-data>yes</collect-hip-data>

I'm Also able to do the same between Workstation and Gateway, but not noticing more than the content of --dump.

When GlobalProtect Windows client is in internal mode, there's no connection to the portal, or the gateway once authentication is done and first hip-report is send.

I only notice one hip-report send to the gateway, when hip-report-interval is reached ( once a hour for me).

I was wondering if this timer is settable and didn't find it, but found this in the gateway configuration: Connection settings > Timeout Configuration: Inactivity Logout: 3 hours ( corresponding to the timeout duration of my userid infos in PAN database. Said 45min upper, but it's in case of using another agent)

with following note:

Users are logged out of GlobalProtect when the gateway does not receive a HIP check from the GlobalProtect app in the specified amount of time.

Seems to me that doing nothing when connection-type=notunnel other than sending hip-reports to the internal gateway when hip-report-interval is reached should cover the need for the internal mode in my case.

I also noticed that when user logoff and GlobalProtect client is closing, it send a notification to the firewall, and the entry for this user is removed from PAN userid database instantly.

[dlenski]

Seems to me that doing nothing when connection-type=notunnel other than sending hip-reports to the internal gateway when hip-report-interval is reached should cover the need for the internal mode in my case.

What happens if you apply this simple patch?

diff --git a/auth-globalprotect.c b/auth-globalprotect.c
index 0b8f19f..101012c 100644
--- a/auth-globalprotect.c
+++ b/auth-globalprotect.c
@@ -234,7 +234,7 @@ static const struct gp_login_arg gp_login_args[] = {
     { .opt="unknown-arg9", .show=1 },
     { .opt="unknown-arg10", .show=1 },
     { .opt="unknown-arg11", .show=1 },
-    { .opt="connection-type", .err_missing=1, .check="tunnel" },
+    { .opt="connection-type", .err_missing=1, .show=1 },
     { .opt="password-expiration-days", .show=1 },   /* days until password expires, if not -1 */
     { .opt="clientVer", .err_missing=1, .check="4100" },
     { .opt="preferred-ip", .save=1 },

This will cause OpenConnect to ignore the notunnel, and just proceed as usual. That means it will subsequently try to pull the gateway config (/ssl-vpn/getconfig.esp). Perhaps this will result in an error, or perhaps the gateway will return a sensible empty config (no routes defined).

If you can build with this patch, run with openconnect --dump -vvvv to get detailed logs, and paste as much of the post-authentication log as possible (starting with POST https://server.company.com/ssl-vpn/getconfig.esp)… that will be very helpful to figure out how to proceed.

[kvernNC]

I compiled with this patch and the merge request with the --force-trojan option.

Version v8.04-8-gc33178cc-dirty de OpenConnect
Using OpenSSL. Features present: TPM (OpenSSL ENGINE not present), PKCS#11, HOTP software token, TOTP software token, DTLS, ESP
Supported protocols: anyconnect (default), nc, gp, pulse

I tried like before to connect directly to the gateway, without talking with the portal: openconnect --protocol=gp --usergroup=gateway 10.0.0.254

response from the gateway, after client POST getconfig.esp is: errors getting SSL/VPN config

Full log is identical to previous attempt until connection-type checking:

...
GlobalProtect login returned authentication-source=(empty_authprofile)
GlobalProtect login returned connection-type=notunnel
POST https://10.0.0.254/ssl-vpn/getconfig.esp
> POST /ssl-vpn/getconfig.esp HTTP/1.1
> Host: 10.0.0.254
> User-Agent: PAN GlobalProtect
> Cookie: PHPSESSID=564960efbb1467d81a9ec14e26047345
> X-Pad: 000000000000000000000000000000000000000
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 281
> 
> client-type=1&protocol-version=p1&app-version=4.0.5-8&clientos=Linux&os-version=linux-64&hmac-algo=sha1%2cmd5%2csha256&enc-algo=aes-128-cbc%2caes-256-cbc&authcookie=474f205348af5daba969ced4807e821f&portal=INTERNAL&user=unsername%40domain.com&domain=domain.int&computer=LAPTOP
Got HTTP response: HTTP/1.1 200 OK
Date: Thu, 29 Aug 2019 22:17:45 GMT
Content-Type: application/xml; charset=UTF-8
Content-Length: 29
Connection: keep-alive
ETag: "23d5c9ab543"
Pragma: no-cache
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 19 Nov 1981 08:52:00 GMT
X-FRAME-OPTIONS: DENY
Strict-Transport-Security: max-age=31536000;
X-XSS-Protection: 1; mode=block;
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline'; img-src * data:; style-src 'self' 'unsafe-inline';
HTTP body length:  (29)
< errors getting SSL/VPN config
Failed to parse server response
Response was:errors getting SSL/VPN config
Creating SSL connection failed

Forcing the Gateway is not moking the same behavior as the official client. What does official client in my case is: POST /global-protect/prelogin.esp >> to the Portal POST /global-protect/getconfig.esp >> to the Portal POST /ssl-vpn/prelogin.esp >> to the Gateway POST /ssl-vpn/login.esp >> to the Gateway

login.esp content is :

prot=https:&
server=10.0.0.10&
inputStr=&
jnlpReady=jnlpReady&
user=username%40domain.com&
passwd=&
computer=LAPTOP&
ok=Login&
direct=yes&
clientVer=4100&
os-version=Microsoft+Windows+10+Enterprise+2016+LTSB+%2c+64-bit&
preferred-ip=&
preferred-ipv6=&
clientos=Windows&
clientgpversion=4.0.8-4&
portal-userauthcookie=empty&
portal-prelogonuserauthcookie=empty&
host-id=fc100246-1a04-4b7a-be02-e94d8285ac3d&
prelogin-cookie=&
ipv6-support=yes&
client-ip=10.0.0.15&
client-ipv6=.

[z3bra]

I started using openconnect and am in the same configuration as @kvernNC, except on an OpenBSD box. I don't have a windows box to provide informations from an official GlobalProtect client, but I will look into setting up a Linux VM to install it.

My configuration is the same: Internal gateway used for identification only (no routes pushed) and 2 external gateways. And could provide another set of logs/dump if needed though as my use case is exactly the same, it might not be truly useful...

I can test building/running on OpenBSD though. I will also try to lookup the code, and see if I can work on a patch (given my limited capabilities).

[z3bra]

From what I could find, it seems the issues comes for the latest call to getconfig.esp, which probably pass incorrect arguments to the gateway.

Currently, with openconnect v8.05 patched to ignore the connection-type attribute, the arguments passed to getconfig.esp are as follows:

client-type=1
&protocol-version=p1
&app-version=4.0.5-8
&clientos=Linux
&os-version=linux-64
&hmac-algo=<URLENCODED-ALGO>
&enc-algo=<URLENCODED-ALGO>
&authcookie=<AUTHCOOKIE>
&portal=<INTERNAL-GATEWAY>
&user=<USERNAME>
&domain=<DOMAIN>
&computer=<COMPUTER>

I think that in the case of an internal gateway, the settings would probably differ, which means that the gateway cannot find a proper config with these parameters. Unfortunately I don't have the official PAN client in my hands to sniff the exact request to send for getconfig.esp.

@kvernNC, would you be able to provide that?

[kvernNC]

Hi, What I am seeing is the last getconfig.esp is not send when connecting to an internal gateway.

I tried to be more detailled here: https://pastebin.com/ptbxu42t

I'm not able to develop in C and I don't understand fully the code globalprotect.c, but I think it needs a different function to handle internal gateway, as the behavior is not the same.

[z3bra]

I can write some C so I'll take a look at auth-globalprotect.c to see how we can handle that. My main concern is that the full process to connect to an internal or external gateway would be totally different, thus needing a complete re-architecture of the connecting process.

Either way, your pastebin is pretty useful :)

[kvernNC]

@z3bra, we decided to stop waiting for a solution with openconnect and we put 802.1X authentication on the switch connecting the few users we have that don't run under windows.

Then, you can send the log of the radius server to the Palo Alto and give him REGEX to retrieve the username and populate his username/ip database, which is what we needed openconnect to do.

Here is the Palo Alto documentation on this.

You should ask your network team if something like that is possible.

[z3bra]

I ended up using the official PanGPS client for linux. It works rather well (though I had issues at first because my domain name wasn't set correctly on my machine).

You were right about the code: it needs a decent architectural change to support different gateway types.

[kvernNC]

Posted on official openconnect gitlab as a feature request for prosperity: https://gitlab.com/openconnect/openconnect/issues/81

[dlenski]

Continuing discussion there. (And I believe you mean “posterity” :grin:)