Unable to access Tunnel webserver

[volitank]

Hello,

I'm trying to implement this software and I'm having some trouble getting started. I was able to register the device and connect to the wireguard tunnel, however I can't seem to get to the Tunnel to register MFA. I'm sure I've configured something improperly. Below is my configuration and the logs from wag

{
    "Socket": "/tmp/wag.sock",
    "Proxied": true,
    "ExposePorts": [
        "443/tcp"
    ],
    "NAT": true,
    "HelpMail": "test@test.com",
    "Lockout": 5,
    "ExternalAddress": "vpn.test",
    "MaxSessionLifetimeMinutes": 1440,
    "SessionInactivityTimeoutMinutes": 60,
    "DownloadConfigFileName": "wg0.conf",
    "ManagementUI": {
        "ListenAddress": "10.10.1.50:4433",
        "Enabled": true
    },
    "Webserver": {
        "Public": {
            "ListenAddress": "10.10.1.50:8080"
        },
        "Tunnel": {
            "Port": "443",
            "CertPath": "/home/admin/public.pem",
            "KeyPath": "/home/admin/private.pem"
        }
    },
    "Authenticators": {
        "DefaultMethod": "totp",
        "Issuer": "Test",
        "Methods": [
            "totp",
            "webauthn"
        ],
        "DomainURL": "https://vpn.test:8080",
        "OIDC": {
            "IssuerURL": "",
            "ClientSecret": "",
            "ClientID": ""
        },
        "PAM": {
            "ServiceName": ""
        }
    },
    "Wireguard": {
        "DevName": "wg0",
        "ListenPort": 5920,
        "PrivateKey": "OMMITED",
        "Address": "10.10.150.1/24",
        "MTU": 1420,
        "DNS": [
            "10.10.1.254/32"
        ]
    },
    "DatabaseLocation": "devices.db",
    "Acls": {
        "Policies": {
            "*": {
                "Allow": [
                    "10.0.0.0/8",
                    "google.com"
                ]
            }
        }
    }
}

admin@wag:~$ sudo ./wag start --config config.json.20240103212135 
2024/01/04 22:04:04 Started firewall management: 
            Setting filter FORWARD policy to DROP
            XDP eBPF program managing firewall
            Allow Iptables FORWARDS to and from wireguard device
            Allow input to VPN host
            Set routing mode to MASQUERADE (NAT)
2024/01/04 22:04:04 Started control socket: 
             /tmp/wag.sock
2024/01/04 22:04:04 Started listening:
            Tunnel Listener:  10.10.150.1:443 
            Public Listener:  10.10.1.50:8080
2024/01/04 22:04:04 Started Managemnt UI:
            Listening: 10.10.1.50:4433
2024/01/04 22:04:04 Wag started successfully, Ctrl + C to stop
2024/01/04 22:04:10 10.10.150.2 endpoint changed 10.10.1.254:58819 -> 10.10.1.254:22097
2024/01/04 22:04:20 http: TLS handshake error from 10.10.150.2:52384: read tcp 10.10.150.1:443->10.10.150.2:52384: i/o timeout
2024/01/04 22:04:25 http: TLS handshake error from 10.10.150.2:52392: read tcp 10.10.150.1:443->10.10.150.2:52392: read: connection reset by peer
2024/01/04 22:04:30 http: TLS handshake error from 10.10.150.2:52393: read tcp 10.10.150.1:443->10.10.150.2:52393: i/o timeout

I have tried enabling proxy, disabling. Adding ports to expose, I did have an MFA section under policies but didn't seem to do anything. When I go to the Tunnel webserver in a browser it just hangs for a long time, looks like it is trying to redirect to register_mfa but never loads anything. I've tried it in firefox, safari and chrome to no avail. I have also tried it without https and that didn't seem to work either. I'm out of Ideas now so I hoping you can point me in the right direction.

[NHAS]

Howdy,

Thanks for all this information!

Could you give me some more details about your network setup, distro that wag is running on and what your client operating system is (Linux, mac, Windows).

I may have vaguely seen this in the past, however we couldn't find the root cause during that time.

Have you tried hitting the endpoint with curl and could you also give me the output of the wag firewall with the cli or the ui

Thanks!

[volitank]

Getting back on trying to troubleshoot this. Our setup isn't too wild.

Client: MacOS 14.1 (23B74) using official WG client from app store Server: AWS EC2 instance running Debian 12 (Bookworm). It's a fresh updated install with just wag. Linux wag 6.1.0-17-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux

I was just doing testing internally, but could be an issue because I was connecting to it from in office, using private IP, which to get to AWS will traverse a Wireguard site-site tunnel. Today I am at home and will see about how it works hitting it from a public IP.

Once I'm able to connect to it remotely I'll see how it acts and get you the information you've requested.

[volitank]

Okay I was able to get it setup remotely. It's pretty much doing the same thing. I'm able to connect to wireguard, but pretty much can't access anything.

Firewall: Not really sure why there are 172 addresses, unless this is a wag thing.

{
    "tester":{
        "Policies":[
            "10.10.1.42/32 policy [mfa(16) any/any]",
            "10.10.1.254/32 policy [public(20) 53/any]",
            "10.10.150.1/32 policy [public(20) any/any]",
            "10.0.0.0/8 policy [public(20) any/any]",
            "172.253.63.100/32 policy [public(20) any/any]",
            "172.253.63.101/32 policy [public(20) any/any]",
            "172.253.63.102/32 policy [public(20) any/any]",
            "172.253.63.113/32 policy [public(20) any/any]",
            "172.253.63.138/32 policy [public(20) any/any]",
            "172.253.63.139/32 policy [public(20) any/any]"
        ],
        "Devices":[
            {"LastPacketTimestamp":0,
             "Expiry":0,
             "IP":"10.10.150.2",
             "Authorized":false
             }
        ],
        "AccountLocked":0
    }
}

Curl doesn't say much other than it is unable to connect to 10.10.150.1 443

[volitank]

I see this happening as well, but I'm not sure if it's relevant. Maybe the db got corrupted? 10.10.1.254 failed to get registration key: sql: no rows in result set

[NHAS]

The 172 addresses may be your dns server if you've set one.

It isn't a wag thing.

As for the registration issue, that just means something is attempting to use a registration token that is no longer valid/has been used.

We also had this issue on a mac device, could you try Linux or another client?

[volitank]

Wow okay so it works from my Linux machine doing it through KDE Network Manager settings. So there is some kind of issue with the official Wireguard client in MacOS then?

[NHAS]

This both surprises me and doesn't surprise me.

Whole the mac wireguard go definitely had some strange behavior I don't think this is caused by that.

I think the crypto libraries on mac might be slightly broken.

When I get back to keyboard I'll make a version for you to try!

[volitank]

Okay sounds great! And yea when I use wireguard normally it does work with this client, just not with wag. I'm looking forward to testing it.

[NHAS]

A bit of a shot in the dark, I've pushed to the tls_changes branch what I hope will fix your issue.

How comfortable are you with compiling wag? I can do it for you if you need

[volitank]

I was able to compile it. Had a little trouble compiling on Debian 12 but I got it done.

Unfortunately though it doesn't seem to change anything.

[volitank]

Seems to me that it may not be Webserver specific. It appears that I'm not really able to reach anything.

Noticing in the Wireguard client that it never shows that the handshake completes. Although in WAG it shows me connecting as it says my endpoint changes.

[volitank]

I have been messing around with this stuff today. And I'm not super knowledgeable with how iptables specifically functions. But it appears that maybe it doesn't clean up rules? maybe you can make this more clear.

admin@wag:~$ sudo iptables --list-rules
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-A INPUT -i wg0 -p tcp -m tcp --dport 8081 -j ACCEPT
-A INPUT -i wg0 -p tcp -m tcp --dport 8081 -j ACCEPT
-A INPUT -i wg0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i wg0 -p icmp -m icmp --icmp-type 8 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wg0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wg0 -j DROP
-A INPUT -i wg0 -p tcp -m tcp --dport 8081 -j ACCEPT
-A INPUT -i wg0 -p tcp -m tcp --dport 8081 -j ACCEPT
-A INPUT -i wg0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i wg0 -p icmp -m icmp --icmp-type 8 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wg0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wg0 -j DROP
-A INPUT -i wg0 -p tcp -m tcp --dport 8081 -j ACCEPT
-A INPUT -i wg0 -p tcp -m tcp --dport 8081 -j ACCEPT
-A INPUT -i wg0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i wg0 -p icmp -m icmp --icmp-type 8 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wg0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wg0 -j DROP
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i wg0 -j ACCEPT
-A FORWARD -o wg0 -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i wg0 -j ACCEPT
-A FORWARD -o wg0 -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i wg0 -j ACCEPT
-A FORWARD -o wg0 -j ACCEPT

I believe that this was initially giving me issues since I was changing configs a lot, stopping and starting it. But I was able to get it to work. Once I rebooted the system it cleared those out and I was able to connect real good and everything works.

The second part that seems to cause trouble is On-Demand in the MacOS client. If that is turned on it seems like it will never even handshake with wireguard, although it does connect to wireguard as the endpoint is updated in the console. Furthermore turning On-Demand off seems to work, but it does look like on Mac I have to start the tunnel, stop the tunnel, and then start it again in order for it to handshake properly.

[volitank]

Also this was on the latest release. I didn't actually need the tls_changes for it to work.

[volitank]

Sorry to be spamming you today. But it seems the connectivity was intermittent. I am unable again to connect on either. I will continue trying to see if I can find a solid way to force it to work. But to note using wg-quick on the MacBook with homebrew it works every time it seems.

[NHAS]

Ah yes. I don't use the wireguard app on mac for deployment reasons but we have the wg-quick work most of the time.

As for on demand.. That's a mac specific feature that I'm not sure how it works. Wag definitely won't play nice with it as it requires the wireguard deamon to be alive to keep the NAT entries the same.

This isn't spamming this is quite useful information. Typically firewall rules aren't cleaned up if wag crashes. Does the wag log give you the "removing firewall rules" on shutdown?

When you say the connectivity is intermittent is it wireguard that's being intermittent or is it that wag is logging you out, which can happen on unstable networks or cg-nat situations

[NHAS]

Glad to hear you got it somewhat working tho!

I'm working on an unstable release that'll fix some performance issues (as best I can as one of the syscalls is slow and there is no way around that yet).

In that release I'll also be changing how iptables does its thing a small bit by blanket allowing icmp which might help.

When I've made the changes (and fixed the tests) I might as you to give it a go!

[volitank]

Ahh yea looks like it is having an issue with iptables. I didn't notice this.

Jan 08 21:07:52 wag wag[413]: 2024/01/08 21:07:52 Got signal interrupt gracefully exiting
Jan 08 21:07:52 wag wag[413]: 2024/01/08 21:07:52 Removing Firewall rules...
Jan 08 21:07:52 wag systemd[1]: Stopping wag.service - Wireguard Manager...
Jan 08 21:07:52 wag wag[413]: 2024/01/08 21:07:52 Unable to clean up firewall rules:  could not get iptables version: signal: interrupt

[NHAS]

Oh wild, the underlying library is failing to do iptables things.

How exactly are you running this again? Just bare on debian? That'll also be breaking basically everything if the rules arent getting inserted properly.

[volitank]

Yea, it's just Debian 12. I'm using the wag.service from the repo.

As for the intermittency, I mean the actual handshaking for Wireguard. When activating the tunnel on the Mac it will never handshake. But it shows endpoint changing in wag logs. Sometimes if I disable and activate on the Mac it will handshake and all is fine. But it doesn't seem very consistent.

[NHAS]

Hmmm I can't really talk to what the mac application does when it comes to wireguard.

Potentially it's only trying to set up the tunnel when you're hitting routes that go through it, which isn't very helpful in this use case.

No idea why debian would be sad, it just means that parsing the iptables output is foaling somehow.

If you could try it on Ubuntu 20.04 that's what I'm running it on.

Or the docker version

[volitank]

Okay some updates. I actually built from the unstable branch just for the hell of it, but it didn't seem to really affect the issue here, and is working fine.

It not being able to clear iptable rules seems to have been caused by my EC2 instance being too small. I jumped up one size and it hasn't failed since. I'm still on Debian 12.

The issues with connecting are definitely some MacOS weirdness. If at any time you run into the situation where it will not connect, if you just leave the tunnel activated and bounce your wifi, it connects fine. So they are doing some sort of caching or something that Wag doesn't play nice with, even without On-Demand enabled.

It actually appears to function exactly the same no matter if On-Demand is enabled or not. I guess the only difference is with On-Demand it can ignore some SSIDs.

Kind of annoying to bounce WiFi, but it's not really any harder than turning the tunnel off and on. For now I think it's usable, I'm going to move forward with authentication stuff and getting a few of our users on it for testing.

Thanks for the help!

[NHAS]

That is incredibly strange.

We run wag using the wireguard-go version from homebrew on about 40 computers and itdoesnt have that issue.

To me it feels like it's not actually sending data through the tunnel from what you've described.

Unfortunately I don't have the time to test the app version of this install for at least a week or two.

But I'll do some more serious debugging later.

I'll be interested to hear how your deployment goes!

[NHAS]

As I cant replicate this, and there isnt much else I can do. I'm going to close this issue for now.