search

trailofbits / algo

Set up a personal VPN in the cloud
https://blog.trailofbits.com/2016/12/12/meet-algo-the-vpn-that-works/
GNU Affero General Public License v3.0
28.87k stars 2.32k forks source link

Unable to connect to websites when connected to VPN #584

Closed donniebishop closed 7 years ago

donniebishop commented 7 years ago

OS / Environment

Ubuntu / DigitalOcean

Ansible version

2.2.0.0

Version of components from requirements.txt

msrestazure: 0.4.8 setuptools: 36.0.1 dopy 0.3.5 boto 2.47.0 boto3 1.4.4 azure 2.0.0rc5 msrest 0.4.1 apache-libcloud 2.0.0 six 1.10.0 pyopenssl 17.0.0 jinja2 2.8

Summary of the problem

After connecting to the VPN via strongswan (ipsec up algo), the connection is successfully established, but I am unable to connect to websites and browse. Pings and mtrs to many sites are successful, both against IPs and domain names (google.com, placekitten.com, and github.com are sites I have been testing). However, browsing via HTTP/HTTPS and connecting to IRC servers fails.

This is only solved by disconnecting from the VPN (ipsec down algo). When disconnecting, I do get an ominous message about an iptables rule:

deleting IKE_SA algo[4] between 172.16.6.21[CN=bushidoboy]...104.236.209.23[104.236.209.23]
sending DELETE for IKE_SA algo[4]
generating INFORMATIONAL request 2 [ D ]
sending packet: from 172.16.6.21[4500] to 104.236.209.23[4500] (65 bytes)
received packet: from 104.236.209.23[4500] to 172.16.6.21[4500] (57 bytes)
parsed INFORMATIONAL response 2 [ ]
IKE_SA deleted
updown: iptables: Bad rule (does a matching rule exist in that chain?).
IKE_SA [4] closed successfully

I did a full iptables flush of all chains, but this did not alleviate the issues I saw.

Steps to reproduce the behavior

Connect to Algo DigitalOcean instance via strongswan (ipsec algo up). Problem is only solved by disconnecting. Only affects my Arch Linux laptop. Issue does not occur on my Android phone when connecting to the VPN

The way of deployment (cloud or local)

Cloud

Expected behavior

Expect to be able to browse and connect to internet sites/irc servers

Actual behavior

I am unable to connect to web and irc servers until I disconnect from my Algo instance

Full log

initiating IKE_SA algo[4] to 104.236.209.23
generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
sending packet: from 172.16.6.21[500] to 104.236.209.23[500] (354 bytes)
received packet: from 104.236.209.23[500] to 172.16.6.21[500] (289 bytes)
parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(HASH_ALG) N(MULT_AUTH) ]
local host is behind NAT, sending keep alives
received cert request for "CN=104.236.209.23"
sending cert request for "CN=104.236.209.23"
authentication of 'CN=bushidoboy' (myself) with ECDSA_WITH_SHA256_DER successful
sending end entity cert "CN=bushidoboy"
establishing CHILD_SA algo
generating IKE_AUTH request 1 [ IDi CERT N(INIT_CONTACT) CERTREQ IDr AUTH CPRQ(ADDR DNS) SA TSi TSr N(MOBIKE_SUP) N(NO_ADD_ADDR) N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
sending packet: from 172.16.6.21[4500] to 104.236.209.23[4500] (897 bytes)
received packet: from 104.236.209.23[4500] to 172.16.6.21[4500] (828 bytes)
parsed IKE_AUTH response 1 [ IDr CERT AUTH CPRP(ADDR DNS DNS) SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_6_ADDR) ]
received end entity cert "CN=104.236.209.23"
  using certificate "CN=104.236.209.23"
  using trusted ca certificate "CN=104.236.209.23"
checking certificate status of "CN=104.236.209.23"
certificate status is not available
  reached self-signed root ca with a path length of 0
authentication of '104.236.209.23' with ECDSA_WITH_SHA256_DER successful
IKE_SA algo[4] established between 172.16.6.21[CN=bushidoboy]...104.236.209.23[104.236.209.23]
installing DNS server 8.8.8.8 via resolvconf
installing DNS server 8.8.4.4 via resolvconf
installing new virtual IP 10.19.48.2
CHILD_SA algo{4} established with SPIs cb7f0147_i cb665213_o and TS 10.19.48.2/32 === 0.0.0.0/0
connection 'algo' established successfully

$ sudo ipsec down algo
deleting IKE_SA algo[4] between 172.16.6.21[CN=bushidoboy]...104.236.209.23[104.236.209.23]
sending DELETE for IKE_SA algo[4]
generating INFORMATIONAL request 2 [ D ]
sending packet: from 172.16.6.21[4500] to 104.236.209.23[4500] (65 bytes)
received packet: from 104.236.209.23[4500] to 172.16.6.21[4500] (57 bytes)
parsed INFORMATIONAL response 2 [ ]
IKE_SA deleted
updown: iptables: Bad rule (does a matching rule exist in that chain?).
IKE_SA [4] closed successfully
jackivanov commented 7 years ago

Did you check the troubleshooting guide?

donniebishop commented 7 years ago

I did. I tried troubleshooting by adjusting the MTU, but I still can't seem to reach sites. Pings go through just fine when connected to my Algo instance, but trying to reach sites via the browser or even via curl and wget fails.

~ $ ip addr show wlp2s0
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1438 qdisc mq state UP group default qlen 1000
    link/ether 78:0c:b8:6c:95:85 brd ff:ff:ff:ff:ff:ff
    inet 172.16.6.21/25 brd 172.16.6.127 scope global dynamic wlp2s0
       valid_lft 316sec preferred_lft 316sec
    inet 10.19.48.1/32 scope global wlp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::63e:9d79:c21b:cf96/64 scope link 
       valid_lft forever preferred_lft forever

~ $ ping www.placekitten.com
PING www.placekitten.com (72.47.195.49) 56(84) bytes of data.
64 bytes from somethinglovely.com (72.47.195.49): icmp_seq=1 ttl=51 time=79.1 ms
64 bytes from somethinglovely.com (72.47.195.49): icmp_seq=2 ttl=51 time=80.1 ms
64 bytes from somethinglovely.com (72.47.195.49): icmp_seq=3 ttl=51 time=80.8 ms
64 bytes from somethinglovely.com (72.47.195.49): icmp_seq=4 ttl=51 time=80.3 ms
64 bytes from somethinglovely.com (72.47.195.49): icmp_seq=5 ttl=51 time=80.6 ms
64 bytes from somethinglovely.com (72.47.195.49): icmp_seq=6 ttl=51 time=79.6 ms
^C
--- www.placekitten.com ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5007ms
rtt min/avg/max/mdev = 79.164/80.156/80.899/0.622 ms

~ $ wget -v https://www.placekitten.com
--2017-06-08 21:09:04--  https://www.placekitten.com/
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving www.placekitten.com... 72.47.195.49
Connecting to www.placekitten.com|72.47.195.49|:443... ^C

The wget was killed after waiting about 2 minutes.

jturner314 commented 7 years ago

I'm experiencing exactly the same symptoms on my Arch Linux laptop, and I also tried adjusting the MTU, which didn't help. The log messages are exactly the same as those posted by @donniebishop except for the IP addresses and username. Like @donniebishop, the issue does not occur on my Android phone.

OS / environment / software versions: same as @donniebishop

On the client machine (my Arch Linux laptop), I'm using strongswan 5.5.3 (built using the AUR package with NetworkManager support enabled with the --enable-nm configure option).

blueonyx commented 7 years ago

i have this problem on my new arch laptop too, but it works fine on my old one.

jturner314 commented 7 years ago

@blueonyx Do you know what's different on your old Arch laptop? (Maybe an older version of strongswan or a different configuration?)

blueonyx commented 7 years ago

i found it's the kernel version!

if i downgrade my new box from linux 4.11.{3,5}-1 to linux 4.10.13-1 it works as expected! likewise if i upgrade linux on my old box to 4.11.5-1 it doesnt work anymore!

although the output of all these commands is the same no matter the kernel version:

ipsec up my-conn ip addr ip route ip route list table 220 ip xfrm policy

jturner314 commented 7 years ago

This does, in fact, appear to be a kernel issue with Linux 4.11:

Edit: By the way, thanks for isolating the issue to the kernel version, @blueonyx! I was messing around with the strongswan version with no success.

donniebishop commented 7 years ago

Nice find @blueonyx! I downgraded to 4.10.10 from 4.11.6 and the VPN is working once more, smooth as silk. Great find, and will certainly be keeping an eye out for updates to the 4.11 strongswan bug - thanks to @jturner314 for finding those links as well :+1:

s-zeng commented 6 years ago

Is it possible to have info about which kernel versions have this problem be put somewhere in the docs?

rdonkin commented 6 years ago

I had a different issue but with very similar symptoms, so I'm commenting here in case it helps (issue was fixed by networking change, see below).

My IPSec connection was from macOS 10.11 to Ubuntu 18.04 (kernel 4.15) server, created in AWS by the algo script.

I found that connecting via one broadband provider worked perfectly. Ping and website access worked for a range of sites.

The other broadband provider (fixed wireless) was fine for pings, and connection stayed up, but only a few websites worked (https://google.com, https://bing.com/), with various http and https sites never finishing load (usually not loading the main page).

I tried restarting the VPN server, reducing MTU to 1300 (ping test was fine), disabling IPv6 (in fact all traffic was IPv4 anyway, this was just to check), etc. I read the Troubleshooting guide but only the MTU action applied.

This provider just did a firmware upgrade for a router (2 hops away from Mac, and not involved in IPSec at all), which completely fixed this.

So if you get this type of issue, I recommend:

Perhaps this could be added to the Troubleshooting doc along with "check you don't have kernel 4.11"?

Algo is now working well - thanks for all the effort that has gone into this!

haroondilshad commented 6 years ago

Is there a work-around for this for a 18.04 Ubuntu client aside from downgrading the kernel? I can't seem to use VPN on Ubuntu but everywhere else it works fine.

Benehiko commented 5 years ago

Same issue here. Ubuntu 18.04 | Kernel Version 4.15 Can establish a connection with the provided strongswan configuration file, but cannot access certain websites through the browser. Can ping them though. On my Android device, it works perfectly.

TC1977 commented 5 years ago

May be an MTU issue as stated above.

https://github.com/trailofbits/algo/blob/master/docs/troubleshooting.md#various-websites-appear-to-be-offline-through-the-vpn

Benehiko commented 5 years ago

Yeah, I tried setting the MTU but didn't work. I redid my configurations and it's working now.

Off topic: I can only connect if I start the connection manually eg. sudo ipsec up <conn-name> or by using auto=start in my ipsec.conf

If I use auto=route in my ipsec.conf file it connects to the VPN server, retrieves an IP, but no internet or connection to other clients on the VPN.