Unable to connect on IPv6-only device (website and app)

[wnhre2ur8cxx8]

Operating system: Debian buster

App version: 2020.7

Issue description

First of: I was not able to download the .deb because you dont provide a AAA record for mullvad.net - I had to download it on another machine and copy it over. So please, please add IPv6 support on your webserver!

Secondly I was unable to connect to the VPN servers. It even fails to create a proper wireguard key because of missing IPv6 support.

Have a look at my logs and please add IPv6 support :pray: I love your service and the professionalism of your work, but this one is a bummer :crying_cat_face:

[2021-01-17 00:31:47.579][mullvad_daemon::version][INFO] Starting mullvad-daemon - 2020.7 2020-11-18
[2021-01-17 00:31:47.579][mullvad_daemon][INFO] Logging to /var/log/mullvad-vpn
[2021-01-17 00:31:47.579][mullvad_daemon::rpc_uniqueness_check][DEBUG] Failed to locate/connect to another daemon instance, assuming there isn't one
[2021-01-17 00:31:47.579][mullvad_daemon][INFO] Management interface listening on /var/run/mullvad-vpn
[2021-01-17 00:31:47.580][mullvad_daemon::relays][DEBUG] Reading relays from /var/cache/mullvad-vpn/relays.json
[2021-01-17 00:31:47.580][mullvad_daemon::relays][DEBUG] Reading relays from /opt/Mullvad VPN/resources/relays.json
[2021-01-17 00:31:47.607][mullvad_daemon::relays][INFO] Initialized with 779 cached relays from 2020-11-18 14:00:57.000
[2021-01-17 00:31:47.607][mullvad_daemon::settings][INFO] Loading settings from /etc/mullvad-vpn/settings.json
[2021-01-17 00:31:47.608][mullvad_daemon::version_check][DEBUG] Loading version check cache from /var/cache/mullvad-vpn/version-info.json
[2021-01-17 00:31:47.608][mullvad_daemon::version_check][WARN] Error: Unable to load cached version info
Caused by: Failed to open app version cache file for reading
Caused by: No such file or directory (os error 2)
[2021-01-17 00:31:47.608][mullvad_daemon::account_history][INFO] Opening account history file in /etc/mullvad-vpn/account-history.json
[2021-01-17 00:31:47.667][mullvad_daemon][DEBUG] No cached target state to load
[2021-01-17 00:31:47.670][talpid_core::routing::imp::imp][DEBUG] Ignored 1 loopback routes
[2021-01-17 00:31:47.671][talpid_core::firewall][INFO] Resetting firewall policy
[2021-01-17 00:31:47.671][talpid_core::firewall::imp][DEBUG] Removing table and chain from netfilter
[2021-01-17 00:31:47.671][mullvad_daemon][INFO] Automatically generating new wireguard key for account
[2021-01-17 00:31:47.671][mullvad_daemon::relays][ERROR] Failed to fetch new relay list: Hyper error. Will retry in 59 minutes
[2021-01-17 00:31:47.672][mullvad_daemon][ERROR] Error: Failed to generate wireguard key
Caused by: Unexpected HTTP request error
Caused by: Hyper error
Caused by: error trying to connect: tcp connect error: Network is unreachable (os error 101)
Caused by: tcp connect error: Network is unreachable (os error 101)
Caused by: Network is unreachable (os error 101)
[2021-01-17 00:31:47.672][mullvad_daemon::management_interface][DEBUG] Broadcasting new wireguard key event
[2021-01-17 00:31:47.674][mullvad_daemon][ERROR] Error: Failed to generate wireguard key
Caused by: Unexpected HTTP request error
Caused by: Hyper error
Caused by: error trying to connect: tcp connect error: Network is unreachable (os error 101)
Caused by: tcp connect error: Network is unreachable (os error 101)
Caused by: Network is unreachable (os error 101)
[2021-01-17 00:31:47.674][mullvad_daemon::management_interface][DEBUG] Broadcasting new wireguard key event
[2021-01-17 00:31:47.675][talpid_core::routing::imp::imp][WARN] Failed to delete routing policy: ip command failed
[2021-01-17 00:33:05.716][mullvad_daemon][ERROR] Error: Failed to generate wireguard key
Caused by: Unexpected HTTP request error
Caused by: Hyper error
Caused by: error trying to connect: tcp connect error: Network is unreachable (os error 101)
Caused by: tcp connect error: Network is unreachable (os error 101)
Caused by: Network is unreachable (os error 101)
[2021-01-17 00:33:05.716][mullvad_daemon::management_interface][DEBUG] Broadcasting new wireguard key event

[faern]

Yes, currently we only connect to our API over IPv4. So uploading a WireGuard key to the API is currently not possible over only IPv6. There are two issues blocking this currently: 1) api.mullvad.net does not even have any IPv6. 2) The app does not rely on DNS, but has a built in mechanism of reaching the API more reliably. But it's IPv4 only currently, so it would need some adjustments.

We are actually currently working on implementing so the app can connect to our WireGuard servers over IPv6. All of them have native IPv6. But as long as the API is only reachable over v4 it won't help that much for your setup I'm afraid.

[wnhre2ur8cxx8]

Good to hear you are already tackling it. So does that mean I could create a wireguard key with the app via IPv4 (like a temporary connection or on another machine) and then use that key to connect via IPv6? Sorry I am not a wireguard expert and I dont know how the app handles all this. Or is this a stupid idea and I should just use bare wireguard with your configs? I wanted to avoid this by using your handy app :smile: Sorry if this question is out of scope in here.

[faern]

Yes hopefully you should be able to do that once we land the WireGuard over IPv6 support. But it's still being developed, so it's nothing you can use right now.

[faern]

We just released app version 2021.3. It has support to connect to WireGuard relays over IPv6. Only via the CLI. But please try it out and see if it floats your boat :)

[wnhre2ur8cxx8]

Wow, thanks for the information. I dont see anything related in the changelog, but I will definitely try it out the next couple of days and report back.

[wnhre2ur8cxx8]

So i figured out how to copy the key (I copied the account_history.json from a working setup) and set my relay specification to: Current constraints: WireGuard over any port over IPv6 in country fi using any provider. I allowed ipv6, and checked for the wireguard key:

Current key    : <my_key>
Key created on : 2021-05-01 17:07:04 +02:00
RPC failed: Failed to verify key
Caused by: Unavailable: Cannot reach the API

Still it wont connect as you can see in the logs here:

[2021-05-02 19:07:49.619][mullvad_daemon::management_interface][DEBUG] connect_tunnel
[2021-05-02 19:07:49.619][mullvad_daemon][DEBUG] Target state Unsecured => Secured
[2021-05-02 19:07:49.620][talpid_core::firewall][INFO] Applying firewall policy: Blocked. Blocking LAN. Allowing endpoint 193.138.218.78:443 over TCP
[2021-05-02 19:07:49.621][mullvad_daemon][DEBUG] New tunnel state: Error(ErrorState { cause: IsOffline, block_failure: None })
[2021-05-02 19:07:49.621][mullvad_daemon][INFO] Blocking all network connections, reason: This device is offline, no tunnels can be established

It just works with the same configuration and key from a machine that has a working IPv4 configuration.

[faern]

Caused by: Unavailable: Cannot reach the API

Yeah this part won't work! The API is still IPv4 only. As noted above. The only thing we added is that the tunnel should be possible to establish over IPv6.

[2021-05-02 19:07:49.621][mullvad_daemon][DEBUG] New tunnel state: Error(ErrorState { cause: IsOffline, block_failure: None })

Ouch. Ok. So our daemon thinks that your device is offline, so it won't even try to connect. The online monitor must have a false negative here. This is not unheard of. I'll assign this to someone with better knowledge of the online monitor.

[pinkisemils]

The offline monitor only checked IPv4 reachability on macOS, since without IPv4, no tunnels could be used anyway. I'll fix this to also check for the existence for IPv6 reachability.

[ghost]

I am using Windows 8.1 and am facing the same problem. I am using OpenVPN instead of Wireguard.

[pavel-odintsov]

I think many things have been changed since this bug was opened.

I use rather unusual IPv6 only setup on my Ubuntu 22.04 and I use my own NAT64 gateway to access IPv4 only Internet.

After I fill my account number in client I see following error: "api.mullwad.net is blocked, please check your firewall'.

Screen:

App version: 2023.3.

In daemon's log file I can clearly see failed attempts to connect to IPv4 addresses which fail as I do not have IPv4 connectivity on my machine:

May 03 12:15:27 station systemd[1]: Started Mullvad VPN daemon.
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_daemon::version][INFO] Starting mullvad-daemon - 2023.3 2023-04-05
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_daemon][INFO] Logging to /var/log/mullvad-vpn
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_daemon::rpc_uniqueness_check][DEBUG] Failed to locate/connect to another daemon instance, assuming there isn't one
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_daemon][INFO] Management interface listening on /var/run/mullvad-vpn
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_api::address_cache][DEBUG] Loading API addresses from /var/cache/mullvad-vpn/api-ip-address.txt
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_api::address_cache][DEBUG] Using API address: 45.83.223.196:443
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_api::availability][DEBUG] Suspending API requests
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_daemon::settings][INFO] Loading settings from /etc/mullvad-vpn/settings.json
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_daemon::version_check][DEBUG] Loading version check cache from /var/cache/mullvad-vpn/version-info.json
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_daemon::version_check][WARN] Error: Unable to load cached version info
May 03 12:15:27 station mullvad-daemon[19789]: Caused by: Failed to open app version cache file for reading
May 03 12:15:27 station mullvad-daemon[19789]: Caused by: No such file or directory (os error 2)
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_relay_selector][DEBUG] Reading relays from /var/cache/mullvad-vpn/relays.json
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_relay_selector][DEBUG] Reading relays from /opt/Mullvad VPN/resources/relays.json
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_relay_selector][INFO] Initialized with 700 cached relays from 2023-04-05 08:54:32.000
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_api::availability][DEBUG] Pausing background API requests
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_daemon::account_history][INFO] Opening account history file in /etc/mullvad-vpn/account-history.json
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_daemon::target_state][DEBUG] No cached target state to load
May 03 12:15:27 station mullvad-daemon[19789]: [talpid_core::firewall][INFO] Resetting firewall policy
May 03 12:15:27 station mullvad-daemon[19789]: [talpid_core::firewall::imp][DEBUG] Removing table and chain from netfilter
May 03 12:15:27 station mullvad-daemon[19789]: [mullvad_api::availability][DEBUG] Unsuspending API requests
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::management_interface][DEBUG] get_account_history
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::management_interface][DEBUG] get_tunnel_state
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::management_interface][DEBUG] get_device
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::management_interface][DEBUG] get_account_history
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::management_interface][DEBUG] get_settings
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::management_interface][DEBUG] get_relay_locations
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::management_interface][DEBUG] get_current_version
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::management_interface][DEBUG] get_current_location
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::management_interface][DEBUG] get_version_info
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon][DEBUG] No version cache found. Fetching new info
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_api::rest][ERROR] Error: HTTP request failed
May 03 12:15:29 station mullvad-daemon[19789]: Caused by: Hyper error
May 03 12:15:29 station mullvad-daemon[19789]: Caused by: error trying to connect: Network is unreachable (os error 101)
May 03 12:15:29 station mullvad-daemon[19789]: Caused by: Network is unreachable (os error 101)
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_relay_selector][INFO] Selected Shadowsocks bridge se-got-br-001 at 185.213.154.117:1236/UDP
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::api][DEBUG] API endpoint: 185.213.154.117:1236
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_api::rest][ERROR] Error: HTTP request failed
May 03 12:15:29 station mullvad-daemon[19789]: Caused by: Hyper error
May 03 12:15:29 station mullvad-daemon[19789]: Caused by: error trying to connect: Network is unreachable (os error 101)
May 03 12:15:29 station mullvad-daemon[19789]: Caused by: Network is unreachable (os error 101)
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_relay_selector][INFO] Selected Shadowsocks bridge se-got-br-001 at 185.213.154.117:1234/UDP
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::api][DEBUG] API endpoint: 185.213.154.117:1234
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_api::rest][ERROR] Error: HTTP request failed
May 03 12:15:29 station mullvad-daemon[19789]: Caused by: Hyper error
May 03 12:15:29 station mullvad-daemon[19789]: Caused by: error trying to connect: Network is unreachable (os error 101)
May 03 12:15:29 station mullvad-daemon[19789]: Caused by: Network is unreachable (os error 101)
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::version_check][ERROR] Failed to fetch version info: Failed to check the latest app version
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon][ERROR] Error: Error running version check
May 03 12:15:29 station mullvad-daemon[19789]: Caused by: Version cache update was aborted
May 03 12:15:29 station mullvad-daemon[19789]: [mullvad_daemon::api][DEBUG] API endpoint: 45.83.223.196:443

The lack of IPv4 connectivity on my machine can be easily confirmed this way:

sudo ip r get 8.8.8.8
RTNETLINK answers: Network is unreachable

I've managed to use Mozilla's VPN over NAT64 gateway but I decided to switch to Mullwad as I clearly see native IPv6 support for VPN servers but client clearly have issues with using IPv6 which is disappointing.

I'll be happy to provide testing or any kind of assistance developers may need as I'm software engineer working in network field.

As I have DNS64 and NAT64 gateway in my network I can see synthesized AAAA records for api.mullvad.net:

dig api.mullvad.net -t aaaa

; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> api.mullvad.net -t aaaa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21382
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;api.mullvad.net.       IN  AAAA

;; ANSWER SECTION:
api.mullvad.net.    56  IN  AAAA    64:ff9b::2d53:dfc1

;; Query time: 160 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Wed May 03 12:36:00 BST 2023
;; MSG SIZE  rcvd: 72

And I can see that curl can easily connect to api.mullwad.net using NAT64 gateway:

curl -v -L https://api.mullvad.net
*   Trying 64:ff9b::2d53:dfc1:443...
* Connected to api.mullvad.net (45.83.223.193) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
....
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=se-mma-api-101.mullvad.net
*  start date: Mar 29 08:02:23 2023 GMT
*  expire date: Jun 27 08:02:22 2023 GMT
*  subjectAltName: host "api.mullvad.net" matched cert's "api.mullvad.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x558294f03eb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: api.mullvad.net
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 404 
< server: nginx
< date: Wed, 03 May 2023 11:37:43 GMT
< content-type: application/json
< content-length: 21
< x-frame-options: DENY
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< referrer-policy: same-origin
< strict-transport-security: max-age=15768000; includeSubDomains
< 
{"code": "NOT_FOUND"}

[faern]

Our app does not use DNS at all to reach api.mullvad.net. for anti-censorship purposes. The app comes bundled with a hardcoded known IP to the API (45.83.223.193) and it opens a connection directly to that IP, no DNS involved. That might be why it does not work in your IPv6-only network.

The app currently definitely need working IPv4 for a lot of stuff, for example API connectivity. This is nothing we currently have on our radar to fix since 1) we do not want to use DNS for anything since it can easily be poisoned and 2) our API server does not even have an IPv6 address.

[pavel-odintsov]

@faern Thank you so much for prompt feedback. Nice trick with DNS use avoidance.

Can I override this IP somehow on my side? Simple replacement to synthetic IPv6 address will work fine in my case: 64:ff9b::45.83.223.193.

Mullwad has exceptionally high value for me as you do support IPv6 towards VPN servers and you have IPv6 support over tunnel.

I'll be fine with limited command line functionality as I do a lot of research in IPv6 field and I'm fine with funny workarounds.

[faern]

Yes and no. There is logic in the app to override the API IP and domain with environment variables. However, for security reasons this is compiled out on release builds. We do this to avoid having users being tricked into pointing their app to a malicious API server.

However, if you install a development build of the app (or compile it yourself) you can change the IP for the API by setting MULLVAD_API_ADDR. See https://github.com/mullvad/mullvadvpn-app/#development-builds-only. Please note that I have never tried this with IPv6 addresses... It might fail horribly. The code was written with IPv4 in mind and nothing else has been tested.