virtio-net / vmnet does not properly NAT packets going out through Cisco VPN

[lmb]

Hi,

I've noticed that xhyve with virtio-net does not properly NAT packets that need to reach a VPNed host.

• The two relevant devices are bridge100 and utun0 (Cisco VPN)
• Packets coming in via bridge100 and leaving to the public internet are properly NATed
• Packets coming in via bridge100 and exiting to utun0 still carry the internal, host-only IP address

I verified this by pinging hosts on the public internet (google.com) and behind the VPN and observing with Wireshark.

Any idea what could be causing this? Where can I look for more information? Finding docs on vmnet.framework is surprisingly difficult.

[testn]

any update to this one? I encountered the same issue

[xez]

Did you file a bug report with Apple?

[digitalresistor]

It sounds like there is a pf rule that needs to be set up for anything leaving into the utun0 tunnel.

[lmb]

Yeah, that's pretty much how far I got as well.

nat pass on utun0 from (bridge100:network) to any -> (utun0)

Will make packets go out properly NATed, return to the interface, but then get dropped somewhere else.

[testn]

Did you notice that if you bring up dlite after you connect to VPN everything is working correctly?

[lmb]

I've done that a few times before, doesn't work. Maybe different types of VPN?

[ariscn]

Any updates on this? Bit of a deal-breaker for using xhyve in a work environment.

[XiaonuoGantan]

Any updates on this? It's preventing me from using xhyve on my dev boxes

[xez]

Docker's xhvye fork seems to have found a way around this issues. Guess we have to wait for their source drop.

[FiloSottile]

I ran into this. I run two VPN, one is a openconnect (not working) and one is a OpenVPN (working).

198.41.0.0/16 is behind openconnect and the rest of the internet is behind OpenVPN.

---

If I ping 8.8.8.8 from the VM, here's a capture of bridge100

and here is the capture of utun1

---

If I ping 198.41.x.x from the VM, here's a capture of bridge100

and here is the capture of utun0

---

The difference is obviously that NAT is happening on utun1 <-> bridge100 but not utun0 <-> bridge100

[FiloSottile]

Got it to work. I followed the instructions here https://kfigiela.github.io/2014/11/07/using-native-os-x-nat-with-virutalbox/ and added the following to my /etc/pf.conf after the rdr-anchor line.

nat on {utun9} proto {tcp, udp, icmp} from {bridge100:network} to any -> {utun9}

I also configured my openconnect script to use interface utun9 and to run sudo pfctl -e -f /etc/pf.conf once connected.

Now, however, the other VPN stopped working -.- I suspect that's because I flushed the pf ruleset while loading these. Interestingly, a docker-machine restart makes the utun0 work again, without losing the utun9 one, which suggest it's adding the rules (without flushing) on startup.

[digitalresistor]

PF on OS X supports anchors and allows scripts to load/unload rules on the fly using the anchors. Find a good anchor and use it along with pfctl, this way you don't break any existing rules or ones added on the fly by various system components.

[FiloSottile]

@bertjwregeer I was just reading http://www.openbsd.org/faq/pf/anchors.html and trying echo "..." | pfctl -a utun9nat -f - but the rules don't seem to apply that way.

What do you mean with "Find a good anchor"?

[lmb]

AFAIK the anchor needs to be rooted in pf.conf, otherwise it is not effective.

Something like nat-anchor yourname, and then add rules to yourname.

You might also be able to attach to the apple internet connection sharing anchor, although I could not get that working.

[digitalresistor]

anchor's have to be defined in your pf.conf, you can't just add a new anchor on the fly.

Apple's default pf.conf specifically comes with nat-anchor "com.apple/*"

That is where all the current NAT rules will be dynamically added (for VM's and the like).

Anchors can be nested too...

You should be able to say:

echo 'anchor utun9nat { ... nat rules here ...}' | pfctl -a com.apple -f -

For example to nest under the existing com.apple anchor. Then later you can flush all rules for your anchor, thereby removing it gracefully from the firewall.

pfctl -a com.apple/utun9nat -F nat

Now your anchor goes away.

Note I haven't tested this on OS X, this is mainly based upon knowledge of anchors on OpenBSD.

[lmb]

Filippo, you said the openvpn utun1 works? Could you post the output of ifconfig for the two tun devices? Maybe open connect is not setting some flag that makes vmnet.framework ignore it.

[FiloSottile]

@bertjwregeer @lmb awesome, thanks! The following "simple" lines work flawlessly now, without interfering with the other VPN.

After connecting, specifying interface utun9 (with hardcoded bridge100 IPs to work even if the docker-machine is down):

RULE="nat on {utun9} proto {tcp, udp, icmp} from 192.168.64.0/24 to any -> {utun9}"
echo "$RULE"  | sudo pfctl -a com.apple/utun9nat -f -

When disconnecting or before connecting to cleanup:

sudo pfctl -a com.apple/utun9nat -F nat

[digitalresistor]

Welcome :-).

[FiloSottile]

@lmb you are right we should still figure out why automatic NAT was missing.

I don't see any difference here

$ ifconfig utun0
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
    inet 10.33.127.240 --> 10.33.127.240 netmask 0xffff0000
$ ifconfig utun9
utun9: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1406
    inet 10.25.134.54 --> 10.25.134.54 netmask 0xffffffff
$ bridge100: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    options=3<RXCSUM,TXCSUM>
    ether 82:e6:50:b1:de:64
    inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255
    Configuration:
        id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
        maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
        root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
        ipfilter disabled flags 0x2
    member: en9 flags=3<LEARNING,DISCOVER>
            ifmaxaddr 0 port 11 priority 0 path cost 0
    Address cache:
        6e:67:2:6d:cf:32 Vlan1 en9 1196 flags=0<>
    nd6 options=1<PERFORMNUD>
    media: autoselect
    status: active
$ ifconfig en9
en9: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
    ether 86:0b:34:08:6c:9b
    media: autoselect
    status: active

Or in the routes

$ netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
0/1                10.33.0.1          UGSc           63       12   utun0
default            192.168.1.1        UGSc            1        0     en0
default            link#12            UCSI            1        0 bridge1
10.25.134/23       10.25.134.54       UGSc            1        0   utun9
10.25.134.54       10.25.134.54       UH            198        0   utun9
10.33/16           10.33.127.240      UGSc          234        0   utun0
10.33.127.240      10.33.127.240      UH              2       20   utun0
10.42.42/24        link#14            UC              4        0 vboxnet
10.42.42.255       link#14            UHLWbI          1        6 vboxnet
46.165.240.174/32  192.168.1.1        UGSc            2        0     en0
108.162.209.36/32  192.168.1.1        UGSc            1        0     en0
127                127.0.0.1          UCS             1        0     lo0
127.0.0.1          127.0.0.1          UH             17    20601     lo0
128.0/1            10.33.0.1          UGSc          171       37   utun0
[ many utun9 routes redacted ]
169.254            link#4             UCS             1        0     en0
192.168.1          link#4             UCS             3        0     en0
192.168.1.1/32     link#4             UCS             2        0     en0
192.168.1.1        30:5a:3a:70:a5:38  UHLWIir         8      278     en0   1030
192.168.1.53       84:d6:d0:10:1f:ec  UHLWIi          2        0     en0    688
192.168.1.179/32   link#4             UCS             1        0     en0
192.168.1.255      link#4             UHLWbI          1        6     en0
192.168.42         192.168.1.1        UGSc            1        0     en0
192.168.64         link#12            UC              3        0 bridge1
192.168.64.6       6e.67.2.6d.cf.32   UHLWIi          6      536 bridge1   1181
192.168.64.255     link#12            UHLWbI          1        1 bridge1
198.41             10.25.134.54       UGSc            1        0   utun9
224.0.0            link#4             UmCS            2        0     en0
224.0.0.251        1:0:5e:0:0:fb      UHmLWI          1        0     en0
255.255.255.255/32 link#4             UCS             1        0     en0

[lmb]

utun9 has a different netmask than utun0. Maybe this makes vmnet think that only the point to point IP is reachable via utun9, and therefore it doesn't bother setting up NAT?

[mowings]

Had the same issue with openvpn not working from within the guest. can confirm this worked for me:

RULE="nat on {tun0} proto {tcp, udp, icmp} from 192.168.64.0/24 to any -> {tun0}"
echo "$RULE"  | sudo pfctl -a com.apple/tun0nat -f -

(replace tun0 with your vpn interface name)

I often use more than a single vpn, though -- so this solution is a little awkward. To really make it seamless, it seems like I'd need to modify my vpn configs to run these commands as scripts after the vpn is connected

[rbellamy]

For what it's worth, this isn't working for me anymore. Same as https://github.com/TheNewNormal/kube-solo-osx/issues/139.

[mowings]

Here's a somewhat more generalized script to get this working here:

https://gist.github.com/mowings/633a16372fb30ee652336c8417091222

This is safe to run automatically when any of your vpns come up -- it flushes existing rules before re-natting

[ChrisBuchholz]

@mowings: Your script seems to work for me.

Will this be fixed in xhyve so the script is not necessary?