Active VPN seems to stop TCP traffic, template VMs completely cut off

[horizon2021]

Qubes OS version:

R4.0

Affected component(s):

Whole System

---

Steps to reproduce the behavior:

Fresh installation with 4.0.1 (two days), but I had issues that felt similar on 3.2 before, onboard nic deactivated - instead PCI Intel nic.

Actual behavior:

all was working in the first place, but with updates and reboots it is the following now: when I start vpn (ovpn via command line) traffic does not pass anymore, the browser does not get the connection, no updates are possible, only ping seems to work.

In the beginning I used the initial VMs (sys-net / sys-firewall) but when the issue started I deleted and created new ones. Since there is no option in the vm creation menu anymore to choose net-vm and proxy-vm's I assume that choosing app-vm which gives network is sufficent. qubes-network and network-manager activated afterwards in the new 'net-vm', also nic assigned afterwards.

It does not matter where I start the vpn connection, in the proxy-vm that also was created as app-vm, or in the vm where I work from, vpn seems to block all traffic only ping works then. however tor traffic does pass in any case. the template vm's are cutt off completely, no matter where I connect them to (net-vm at the lowest) they will not get updates, but ping is still possible.

General notes:

in the process I aslo tried: systemctl enable NetworkManager-dispatcher.service but no improvement from that.

---

Related issues:

[horizon2021]

sorry, I have to correct one thing. the the templates vm's do have connection like all other vm's too. the update process appears to have some issue, but that may or may not be not connected with the vpn problem. so I can connect with them to the internet but when I activate vpn (on the proxy-vm) the connection is blocked thereafter (for the template), getting through the vpn via whonix however works and vm's then connected after whonix will have internet access again.

the debian template currently gives this output when I try to update via a working connection (from sys-net):

user@debian-9:~$ sudo apt-get update
Ign:1 https://deb.debian.org/debian stretch InRelease
Ign:2 https://deb.debian.org/debian-security stretch/updates InRelease
Ign:3 http://deb.qubes-os.org/r4.0/vm stretch InRelease
Ign:4 https://deb.debian.org/debian jessie-backports InRelease
Err:5 https://deb.debian.org/debian stretch Release
  Bad header line
Err:6 http://deb.qubes-os.org/r4.0/vm stretch Release
  Connection failed
Err:7 https://deb.debian.org/debian-security stretch/updates Release
  Bad header line
Err:8 https://deb.debian.org/debian jessie-backports Release
  Bad header line
Reading package lists... Done
E: The repository 'https://deb.debian.org/debian stretch Release' does no longer have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://deb.qubes-os.org/r4.0/vm stretch Release' does no longer have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'https://deb.debian.org/debian-security stretch/updates Release' does no longer have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'https://deb.debian.org/debian jessie-backports Release' does no longer have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
user@debian-9:~$ 

and this output when connected from a proxy-vm with active vpn (it halts for about half a minute at Err:6):

user@debian-9:~$ sudo apt-get update
Ign:1 https://deb.debian.org/debian stretch InRelease
Ign:2 https://deb.debian.org/debian-security stretch/updates InRelease
Ign:3 https://deb.debian.org/debian jessie-backports InRelease
Err:4 https://deb.debian.org/debian stretch Release
  Bad header line
Err:5 https://deb.debian.org/debian-security stretch/updates Release
  Bad header line
Err:6 https://deb.debian.org/debian jessie-backports Release
  Bad header line
Ign:7 http://deb.qubes-os.org/r4.0/vm stretch InRelease
Err:8 http://deb.qubes-os.org/r4.0/vm stretch Release
  Connection failed
Reading package lists... Done
E: The repository 'https://deb.debian.org/debian stretch Release' does no longer have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'https://deb.debian.org/debian-security stretch/updates Release' does no longer have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'https://deb.debian.org/debian jessie-backports Release' does no longer have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://deb.qubes-os.org/r4.0/vm stretch Release' does no longer have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
user@debian-9:~$ 

[horizon2021]

So I did another reinstall to have a closer look. Everything I describe now is from the first start of the system, no reboot, no update to dom0 and no update to a template that would affect sys-net or sys-firewall (both fedora as by default and created by the installation scripts, no services are active, not on sys-net not on sys-firewall, ie I use them as created in the first place).

sys-net connected to the internet, sys-firewall does the vpn. when I use a disposable-vm or some of the created ones (work-vm / fedora), activating the vpn on sys-firewall will block all webtraffic, update will respond like this:

[user@work ~]$ sudo dnf update
Fedora Modular 29 - x86_64                      0.0  B/s |   0  B     00:20    
Error: Failed to synchronize cache for repo 'fedora-modular'
[user@work ~]$ 

wget gives this:

[user@work ~]$ wget google.com
--2019-02-20 19:18:52--  http://google.com/
Resolving google.com (google.com)... failed: Name or service not known.
wget: unable to resolve host address ‘google.com’
[user@work ~]$ 

only ping works (as described before).

the templates react differently, I can get through with active vpn on sys-firewall for updates or installations. ping also ok (as before), but the browser will not get anything, same as wget:

user@debtest:~$ wget google.com
--2019-02-20 18:16:40--  http://google.com/
Resolving google.com (google.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘google.com’
user@debtest:~$

[tasket]

I have doubts about this being a bug.

@horizon2021 Did you check iptables -t nat -L to see if the PR-QBS chain has been updated with your DNS addresses? This is a requirement for DNS to be routed over a tunnel on Qubes (on edit: meaning in a provides-networking appVM).

See the Qubes vpn doc for a scripted example, or you can use solution like Qubes-vpn-support.

[simonbytes]

Because you posted some apt_get stuff for connection checking (!?), I'll write it the easy way:

Ping should not work for any unprivileged user in your proxy_vpn qube. It is safer to forbid connections from/to inside your proxy_vm and only allow forwarded traffic.

Check if cat /proc/sys/net/ipv4/ip_forward is set to 1. Otherwise no traffic is forwarded and depending on your iptables settings, no (some) traffic is allowed to leave qubes_vm either.

Check /etc/resolv.conf on your proxy_vpn after initialising ovpn. Should be your vpn dns server, if not using another script. Most external scripts for openvpn change only the /etc/resolv.conf nameserver file. In tasket's qubes-vpn-support openvpn-script (see link below or tasket's post) DNS traffic is rerouted with iptables to the dns server of your vpn. Check sudo iptables -t nat -S PR-QBS to see where DNS traffic is routed to. It does not matter if you set another simple (=port 53) dns nameserver in your AppVMs connected to your vpn proxy_vm. tasket's qubes_vpn_support script is using the iptables PR-QBS table.

Check route -n on your vpn_proxy_vm after initialising ovpn. Should be your internal vpn IP at first with destination 0.0.0.0 and UG flag and your qubes interface uplink IP at second with the same destination and UG flag. The external (internet) vpn IP is routed via your qubes interface uplink IP in another rule in your route table.

p.s.: If you post such command results in here, do not forget to change the ip numbers + - ~42 (or any number). Do not post sensitive data.

Otherwise, I support tasket's message to use Qubes-vpn-support. Read their README.

Out of topic: In the Qubes vpn doc -> section Troubleshooting, the use of sudo sg qvpn (in my example sudo sg qvpn 'ping -c 1 8.8.8.8') is not permitted while using tasket's qubes-vpn-support. This could be less helpful for inexperienced users to solve problems.

@tasket What do you think what other commands are useful to clear problematic situations with vpn's in qubes? So user can check with these commands some misconfigurations on their own, even if it is written dozen of times in other vpn howto's. Could be added to the vpn doc of qubes. And would not it be an idea to use socat with TLS encryption to encrypt port 53 dns traffic to a port 853 dns server on the fly && transparent with qubes-vpn-support if the user sets a DNSSEC server (Port 853) via env vpn_dns string? (Information for those who do not know DNSSEC: https://en.wikipedia.org/wiki/DNS_over_TLS) More Info for DNS over TLS: https://www.rfc-editor.org/rfc/rfc7858.txt

[horizon2021]

Thanks @tasket @simonbytes for the substantial replies.

I did notice that the resolver configuration is sort of an issue and has the be right for a functioning vpn, but it worked almost always well with Debian in 3.2 with the simple ovpn command and under the assumption that it should be executed in the same way in 4.0, thus be working, I put my issue here.

Meanwhile I have deleted the original sys-net, sys-firewall, created other network vm's with different names, deleted again and recreated with the original names: sys-net, sys-firewall.

Now, from sys-net the vpn did work right away (gives network to all connected vm's), from sys-firewall still the same problem (dom0 has gotten no updates and none for the image of the network vm's except ovpn installed for debian), but it is likely a vpn issue that I will have to fix.

Thanks for the remark on private data, I will keep that in mind (but not so super relevant for me right now).

cat /proc/sys/net/ipv4/ip_forward

is 1 on both networking vm's - just to check, the app vm's give me back 0

that the vpn connection now works from sys-net seems to be related to this:

user@sys-net:~$ cat /etc/resolv.conf
# Generated by NetworkManager
search xxx.xxx (my router)
nameserver xxx.xxx.xxx.x (local IP)

there is a online tool that I can use to test my connection and like it is now, it shows IPv6 traffic and some part of the home network that was otherwise not be listed there before (as with 3.2) - it also says I use my original providers dns

sys-firewall with active vpn gives this:

user@sys-firewall:~$ cat /etc/resolv.conf
nameserver 10.139.1.1
nameserver 10.139.1.2

user@sys-firewall:~$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
PR-QBS     all  --  anywhere             anywhere            
PR-QBS-SERVICES  all  --  anywhere             anywhere            

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
MASQUERADE  all  --  anywhere             anywhere            

Chain PR-QBS (1 references)
target     prot opt source               destination         
DNAT       udp  --  anywhere             10.139.1.1           udp dpt:domain to:10.139.1.1
DNAT       tcp  --  anywhere             10.139.1.1           tcp dpt:domain to:10.139.1.1
DNAT       udp  --  anywhere             10.139.1.2           udp dpt:domain to:10.139.1.2
DNAT       tcp  --  anywhere             10.139.1.2           tcp dpt:domain to:10.139.1.2

Chain PR-QBS-SERVICES (1 references)
target     prot opt source               destination         

user@sys-firewall:~$ sudo iptables -t nat -S PR-QBS
-N PR-QBS
-A PR-QBS -d 10.139.1.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.139.1.1
-A PR-QBS -d 10.139.1.1/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 10.139.1.1
-A PR-QBS -d 10.139.1.2/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.139.1.2
-A PR-QBS -d 10.139.1.2/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 10.139.1.2

user@sys-firewall:~$ sudo route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.7.3.1        128.0.0.0       UG    0      0        0 tun0
0.0.0.0         10.137.0.5      0.0.0.0         UG    0      0        0 eth0
10.7.3.0        0.0.0.0         255.255.255.0   U     0      0        0 tun0
10.137.0.5      0.0.0.0         255.255.255.255 UH    0      0        0 eth0
128.0.0.0       10.7.3.1        128.0.0.0       UG    0      0        0 tun0
185.244.212.59  10.137.0.5      255.255.255.255 UGH   0      0        0 eth0

I will move to Qubes-vpn-support.

[simonbytes]

1. In Qubes, you should not use sys-net with vpn software. The idea behind qubes is to separate software, internet connectivity and sensitive data in separate virtual operating systems. You are using the vpn software in sys-net (which would be the first layer for any attacks, because it is directly connected to a network not only through software but also through hardware (probably restricted and patented) and their driver (probably open source at best) and therefore, if any breaches occur, routes can be changed or vpn information could be used. Even MITM attacks are possible etc pp

You should create an AppVM in Qubes 4 and change with qvm-prefs in dom0 the provides_network setting to True of that qube. qvm-prefs --help for more information. Clone your vpn AppVM to play around, without breaking your finished/preconfigured vpn settings.

And try to use tasket's recommended software in (another) AppVM with provides_network setting.

And 2.: ip_forward is automatically set in Qubes if connected. The other informations seems correct, except your routes show the main problem: You are using an active vpn connection within your firewall qube (as already said, not a good idea. Not even for testing); all connections are routed you can see in the list. Your dns server's network (10.139.1.0) for example is not available in the routes, the dns server can not be reached, thus no domain name resolution. You should at least set the dns server your vpn service provides. (or some service on the internet) Without dns, you can test pinging ip adresses: 8.8.8.8, 1.1.1.1

edit: You said in your first post:

...however tor traffic does pass in any case. the template vm's are cutt off completely, no matter w...

Tor is using ip adresses to connect. (I think) So no dns necessary. That indicates the problem is your configuration and probably not a bug in qubes.

[horizon2021]

Installing 'resolvconf' helped. VPN connection will work if you install this for the VM that you visit the iInternet with, installed in a 'net-vm' the situation remains unchanged. Thanks again.

[andrewdavidwong]

Installing 'resolvconf' helped. VPN connection will work if you install this for the VM that you visit the iInternet with, installed in a 'net-vm' the situation remains unchanged. Thanks again.

So, what's the action here?

1. Do we need to make sure resolvconf gets installed for VPN users?
2. Do we need to document this?
3. Did this turn out to be not a bug or invalid?

What do you think, @tasket?

[tasket]

I don't think resolvconf is necessary; I think if you asked Patrick he'd even say it should be avoided since it was his request that prompted me to forgo updating /etc/resolv.conf.

DNAT in the VPN VM is necessary, and from what I can tell that hasn't been satisfied. In horizon2021's case the sys-firewall iptables shows the default Qubes DNAT, not updated for the VPN's DNS servers.

Putting resolvconf in the downstream appVMs ("the VM that you visit the iInternet with") is an interesting approach, but relying on it means you're still using a broken setup where the VPN VM doesn't have the ability to re-address all DNS requests to the VPN's DNS server; since appVMs may be malicious/untrusted, using this workaround is a potential security and privacy issue.

Documentation could be difficult, because you'd be describing to the user how to recognize their VPN provider's DNS IPs.

The cause could be as simple as not setting +x on 'qubes-vpn-handler.sh', so probably not a bug or invalid due to error-prone install process and need to use the mailing list instead. The best solution to avoid these problems is using a single installer like Qubes-vpn-support.

[andrewdavidwong]

I don't think resolvconf is necessary; I think if you asked Patrick he'd even say it should be avoided since it was his request that prompted me to forgo updating /etc/resolv.conf.

DNAT in the VPN VM is necessary, and from what I can tell that hasn't been satisfied. In horizon2021's case the sys-firewall iptables shows the default Qubes DNAT, not updated for the VPN's DNS servers.

Putting resolvconf in the downstream appVMs ("the VM that you visit the iInternet with") is an interesting approach, but relying on it means you're still using a broken setup where the VPN VM doesn't have the ability to re-address all DNS requests to the VPN's DNS server; since appVMs may be malicious/untrusted, using this workaround is a potential security and privacy issue.

Documentation could be difficult, because you'd be describing to the user how to recognize their VPN provider's DNS IPs.

The cause could be as simple as not setting +x on 'qubes-vpn-handler.sh', so probably not a bug or invalid due to error-prone install process and need to use the mailing list instead. The best solution to avoid these problems is using a single installer like Qubes-vpn-support.

Ok, closing as "not an issue." If anyone believes this is a mistake or would like this to be reopened for some reason, please leave a comment, and we'll be happy to take another look. Thank you.