search

pieterlange / kube-openvpn

:closed_lock_with_key: Kubernetes native OpenVPN
MIT License
383 stars 57 forks source link

Connectivity issues on cluster created by Rancher #64

Closed fabioroger closed 6 years ago

fabioroger commented 6 years ago

Have you tested this in a non-gke environment?

Once my client connects to server, I can't access anything other than the openvpn pod.

Communication between client and the openvpn pod seems fine, but can't connect from client to anything else (in kubernetes cluster or not)

I tried changing OVPN_NETWORK and also played with OVPN_DEFROUTE, but of no avail. Also took a look on debug logs but no obvious errors show up.

Any ideas?

Initialization logs:

Fri Aug 24 18:03:59 2018 Running 'openvpn --config /etc/openvpn/openvpn.conf --push route 10.43.0.0 255.255.0.0 --push route 10.42.0.0 255.255.0.0 --client-config-dir /etc/openvpn/ccd --crl-verify /etc/openvpn/crl/crl.pem --status /etc/openvpn/status/server.status --status-version 2 '
Fri Aug 24 18:03:59 2018 OpenVPN 2.4.4 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Nov  9 2017
Fri Aug 24 18:03:59 2018 library versions: LibreSSL 2.6.3, LZO 2.10
Fri Aug 24 18:03:59 2018 Diffie-Hellman initialized with 2048 bit key
Fri Aug 24 18:03:59 2018 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri Aug 24 18:03:59 2018 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri Aug 24 18:03:59 2018 TUN/TAP device tun0 opened
Fri Aug 24 18:03:59 2018 TUN/TAP TX queue length set to 100
Fri Aug 24 18:03:59 2018 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Fri Aug 24 18:03:59 2018 /sbin/ip link set dev tun0 up mtu 1500
Fri Aug 24 18:03:59 2018 /sbin/ip addr add dev tun0 10.140.0.1/24 broadcast 10.140.0.255
Fri Aug 24 18:03:59 2018 Routing 10.42.0.36:20080 to 10.140.0.5:8000 (fabio)
Fri Aug 24 18:03:59 2018 Could not determine IPv4/IPv6 protocol. Using AF_INET
Fri Aug 24 18:03:59 2018 Socket Buffers: R=[87380->87380] S=[16384->16384]
Fri Aug 24 18:03:59 2018 Listening for incoming TCP connection on [AF_INET][undef]:1194
Fri Aug 24 18:03:59 2018 TCPv4_SERVER link local (bound): [AF_INET][undef]:1194
Fri Aug 24 18:03:59 2018 TCPv4_SERVER link remote: [AF_UNSPEC]
Fri Aug 24 18:03:59 2018 GID set to nogroup
Fri Aug 24 18:03:59 2018 UID set to nobody
Fri Aug 24 18:03:59 2018 MULTI: multi_init called, r=256 v=256
Fri Aug 24 18:03:59 2018 IFCONFIG POOL: base=10.140.0.2 size=252, ipv6=0
Fri Aug 24 18:03:59 2018 MULTI: TCP INIT maxclients=1024 maxevents=1028
Fri Aug 24 18:03:59 2018 Initialization Sequence Completed

Client connecting logs:

Fri Aug 24 18:05:18 2018 TCP connection established with [AF_INET]X.X.X.X:54972
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 TLS: Initial packet from [AF_INET]X.X.X.X:54972, sid=e96080a4 ca7d61d1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 VERIFY OK: depth=1, CN=XXXX
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 VERIFY OK: depth=0, CN=fabio
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_VER=2.4.6
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_PLAT=mac
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_PROTO=2
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_NCP=2
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_LZ4=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_LZ4v2=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_LZO=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_COMP_STUB=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_COMP_STUBv2=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_TCPNL=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_GUI_VER="net.tunnelblick.tunnelblick_5080_3.7.6a__build_5080)"
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES128-GCM-SHA256, 2048 bit RSA
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 [fabio] Peer Connection Initiated with [AF_INET]X.X.X.X:54972
Fri Aug 24 18:05:19 2018 fabio/X.X.X.X:54972 OPTIONS IMPORT: reading client specific options from: /etc/openvpn/ccd/fabio
Fri Aug 24 18:05:19 2018 fabio/X.X.X.X:54972 MULTI: Learn: 10.140.0.5 -> fabio/X.X.X.X:54972
Fri Aug 24 18:05:19 2018 fabio/X.X.X.X:54972 MULTI: primary virtual IP for fabio/X.X.X.X:54972: 10.140.0.5
Fri Aug 24 18:05:20 2018 fabio/X.X.X.X:54972 PUSH: Received control message: 'PUSH_REQUEST'
Fri Aug 24 18:05:20 2018 fabio/X.X.X.X:54972 SENT CONTROL [fabio]: 'PUSH_REPLY,block-outside-dns,dhcp-option DOMAIN svc.cluster.local,dhcp-option DNS 10.43.0.10,route 10.43.0.0 255.255.0.0,route 10.42.0.0 255.255.0.0,route-gateway 10.140.0.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.140.0.5 255.255.255.0,peer-id 0,cipher AES-256-GCM' (status=1)
Fri Aug 24 18:05:20 2018 fabio/X.X.X.X:54972 Data Channel: using negotiated cipher 'AES-256-GCM'
Fri Aug 24 18:05:20 2018 fabio/X.X.X.X:54972 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Fri Aug 24 18:05:20 2018 fabio/X.X.X.X:54972 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key

netstat -rn:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         169.254.1.1     0.0.0.0         UG        0 0          0 eth0
10.140.0.0      0.0.0.0         255.255.255.0   U         0 0          0 tun0
169.254.1.1     0.0.0.0         255.255.255.255 UH        0 0          0 eth0

iptables-save:

# Generated by iptables-save v1.6.1 on Fri Aug 24 18:09:38 2018
*nat
:PREROUTING ACCEPT [129:9276]
:INPUT ACCEPT [1:60]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBEOPENVPNPORTFORWARD - [0:0]
-A PREROUTING -j KUBEOPENVPNPORTFORWARD
-A POSTROUTING -s 10.140.0.0/24 -d 10.43.0.0/16 -o eth0 -j SNAT --to-source 10.42.0.36
-A POSTROUTING -s 10.140.0.0/24 -d 10.42.0.0/16 -o eth0 -j SNAT --to-source 10.42.0.36
-A KUBEOPENVPNPORTFORWARD -d 10.42.0.36/32 -p tcp -m tcp --dport 20080 -j DNAT --to-destination 10.140.0.5:8000
COMMIT
# Completed on Fri Aug 24 18:09:38 2018
pieterlange commented 6 years ago

Check the following:

How are you checking connectivity?

fabioroger commented 6 years ago

First, thank you very much for your reply.

So,

For pods CIDR:

$ kubectl cluster-info dump | grep -i cidr
                "PodCIDR": "10.42.0.0/24",
        "subnet": "usePodCidr"

For services CIDR, since all start with 10.43 and assumed their CIDR to be 10.43.0.0/16.

As for NetworkPolicies, I'm not at all familiar with them, but:

$ kubectl get NetworkPolicy --all-namespaces
No resources found.

And I'm checking connectivity with netcat. Calls like this just hang forever:

$ nc -v 10.42.0.10 53

against all known services/pod ips and ports. The only ports I can reach are the ones in openvpn own pod.

Also, for sanity check, I just redid the steps against a Google Cluster cluster and everything works perfectly.

Again thank you for your help. Any other ideas would be greatly appreciated.

fabioroger commented 6 years ago

Found the problem!

net.ipv4.ip_forward was 0 for openvpn container.

Running this on the node did the trick:

$ nsenter -t $(docker inspect --format '{{.State.Pid}}' $CONTAINER_NAME) -n sysctl -w net.ipv4.ip_forward=1