Some sites and apps doesnt work anymore with mullvad active

[ChuckNorrison]

Issue report

Operating system: Ubuntu 22.04

App version: 2022.3

Issue description

Some sites does not work anymore in example github sign in and dropbox desktop app cant sync. Some other sites like duckduckgo.com work.

Tested with wireguard and openvpn settings, with and without ipv6.

Was able to solve the issue with downgrade to 2022.2

[faern]

Did you try connecting to multiple different VPN servers? When a site does not work, the most common issue is that that site blocks our exit IP(s). Changing servers can sometimes help. If you use WireGuard, connecting to the same server after downgrading is sadly not a good way to rule out this possibility either. Because each server we have have multiple exit IPs, and you get a random one depending on your WireGuard key in use. So after your downgrade you have a different WireGuard key, and will probably exit through a different IP even when connecting to the same server.

However, if you had this issue on OpenVPN also, and you have it on many/all servers, then it could of course be an app issue. We'll try it. However, given the amount of users who already use 2022.3, I think we would have gotten flooded with issue emails if it affected everyone/most people.

[ChuckNorrison]

Hey, thanks for your response i tested a lot of servers, at least 20 different connections used, but was not able to get it work. After quit app everything was fine. After uninstall 2022.3 and install 2022.2, everything was fine again with active wireguard connections.

Iam using mullvad several months without issues and updated to 2022.3 today and got this behaviour.

[felschr]

I've experienced the same issue starting with 2022.3 on NixOS. I think it might be IPv6 related, but I'm not entirely sure.

It might be worth noting, that I set my system up to prefer IPv6 via /etc/gai.conf:

label ::1/128       0
label ::/0          1
label 2002::/16     2
label ::/96         3
label ::ffff:0:0/96 4

For now, I've rolled back to 2022.2, which is working fine for me.

Related NixOS issue: https://github.com/NixOS/nixpkgs/pull/186743 (I posted some logs there)

[faern]

@ChuckNorrison What's your setup with regards to IPv6? Anything special going on?

[ChuckNorrison]

@faern tested with ipv6 disabled, did not solved the issue for me.

[faern]

@felschr @ChuckNorrison Do you have more examples of sites that do not work? Would be easier to test if we have a bunch of sites not requiring apps or logins to test.

[ChuckNorrison]

@faern This trading engine from bitmex stopped working. In case of failure a pop up will be displayed with a reconnect progress bar

https://www.bitmex.com/app/trade/XBTUSD

[faern]

I cannot reproduce. That page loads perfectly for me while connected to a bunch of different servers over WireGuard using version 2022.3 on Fedora Linux. I have tried both with and without a native IPv6 connection on the machine and with and without enabling IPv6 in the WireGurad tunnel also.

EDIT: I tried with both Firefox and Chrome

[ChuckNorrison]

@faern maybe an kernel compatibility issue? Ubuntu 22.04 is running on 5.15.0

[faern]

Which web browser are you using? Could you maybe open the development tools in that browser and try to see what kind of connections it tries to make, and even better, try to identify which ones fail.

[ChuckNorrison]

@faern have tested several browsers. Firefox, brave and opera. In case of bitmex the websocket connection was missing, it timed out. Also spotted some CORS failures in the console on github.

[faern]

@faern tested with ipv6 disabled, did not solved the issue for me.

Disabled it how? You disabled IPv6 on your physical network adapter or in our app? If in our app did you disable "Advanced -> Enable IPv6" OR is it the "Advanced -> WireGuard settings -> IP Version" setting? The first one controls if the tunnel should allow carrying IPv6 in it. The latter is if we should connect to our VPN server using IPv6 or v4.

[ChuckNorrison]

@faern tested with ipv6 disabled, did not solved the issue for me.

Disabled it how? You disabled IPv6 on your physical network adapter or in our app? If in our app did you disable "Advanced -> Enable IPv6" OR is it the "Advanced -> WireGuard settings -> IP Version" setting? The first one controls if the tunnel should allow carrying IPv6 in it. The latter is if we should connect to our VPN server using IPv6 or v4.

I used the "Advanced -> Enable IPv6" in app to disable.

[faern]

Maybe we are chasing a red herring here. But do you have native IPv6 connectivity on your physical interface / from your ISP?

[ChuckNorrison]

Maybe we are chasing a red herring here. But do you have native IPv6 connectivity on your physical interface / from your ISP?

Yes, IPv6 is active per default without any VPN enabled

[faern]

I have now tried Ubuntu 22.04 with and without IPv6 on my physical interface. The bitmex site above just works. If by works I see a bunch of flashing red and green and prices are updated in real time :)

It might be worth noting, that I set my system up to prefer IPv6 via /etc/gai.conf:

label ::1/128       0
label ::/0          1
label 2002::/16     2
label ::/96         3
label ::ffff:0:0/96 4

These are the default settings. At least in Ubuntu 22.04. So explicitly setting them makes no difference for me.

I'm sorry, but I really can't reproduce this. You can help me by listing more sites that does not work and possibly by trying to figure out what other special settings you might have on your computer.

[felschr]

A lot of websites only loaded partially for me, even if they seem to be fine at first glance. Some pages just load endlessly. Also, it seemed to change over time: Sometimes github.com didn't load at all while other times only some requests failed.

I haven't tested a lot of pages, but I had this issue with github.com, trakt.tv, calendar.proton.me & app.element.io. Element looked fine, but sending messages didn't work. Proton Calendar showed some error that calendar events couldn't be loaded. A large amount of pages that I visited were affected, but I can't remember any other specific examples right now. github.com & trakt.tv don't require logins to test.

I tried it with Firefox (w/ and w/o SOCKS proxy) & Chromium. I'm experiencing the issues in all of these setups, but sometimes pages partially work in one browser while they just load endlessly in the other.

I have native IPv6 from my provider and IPv4 only via DS-Lite.

[faern]

Could it maybe be a an MTU issue? Sadly we have a bug in 2022.3 that does not allow you to set the MTU from within our app directly. This is going to be fixed. But until then, can you please try by lowering the tunnel MTU manually with this command:

sudo ip link set dev wg-mullvad mtu 1280

Please mind that you need to re-run this command every time a new tunnel is established. This only works on WireGuard tunnels, so try with that.

A too large MTU usually leads to some random resources not loading. Because they are so large that they get fragmented, and some upstream router drops fragmented packets.

[faern]

If you do have native IPv6 and your only IPv4 connectivity is via DS-Lite, you might want to consider establishing the VPN tunnel over IPv6 directly, to completely avoid the IPv4-in-IPv6 that DS-Lite consists of. To do that, either navigate "Advanced -> WireGuard settings -> IP version" in the GUI, or run this command:

mullvad relay set tunnel wireguard --ipv 6

In the future we'll make the app better detect the presence of IPv6 and start using it automatically if preferred by the OS. But currently we always use IPv4 unless explicitly requested otherwise by the app user. This is partially also because IPv6 peering is so crappy on large parts of the internet, so even users with native IPv6 can often benefit from using IPv4 sadly.

[felschr]

I've now been running 2022.3 for a while with the suggested MTU change and the networking issues seem to be gone for me. I also tried your IPv6 suggestion, but with that setting Mullvad can't connect to servers anymore.

So, perhaps the MTU frontend bug you mentioned also causes different default values? I never changed the MTU setting in Mullvad before.

[felschr]

I just realised, the changelog actually mentions

Automatically attempt to detect and set the correct MTU for Wireguard tunnels.

I'm guessing this new procedure sets a higher MTU than it previously did on my machine.

[felschr]

I've just tried it. With 2022.3 wg-mullvad's MTU is set to 1420, while it's 1380 with 2022.2. The MTU of my enp39s0 network is 1500.

[faern]

Thank you for helping us find this! IPv6 was a red herring all along.

We'll look into this and probably issue a new release fairly soon then.

[kubrickfr]

I can reproduce the issue on my mobile ISP.

The issue is that mullvad does Path MTU discovery wrong. All it does is listing interfaces and getting their MTU instead of doing real Path MTU discovery uising UDP (which is very hard to implement correctly).

So, as I was saying, on my mobile ISP, I get an interface MTU of 1500, and I can use the whole of it in IPv6, but my IPv4 connectivity is actually tunnelled over IPv6 and therefore I get a lot less and mullvad MTU detection only works when IPv6 is used for tunnel transport.

[faern]

Please try this new release: https://github.com/mullvad/mullvadvpn-app/releases/tag/2022.4

[felschr]

The release works for me, and the MTU is now 1380 again. Thank you for resolving this so quickly 🥳.

[faern]

Awesome! Then we can close this I think.