search

kylemanna / docker-openvpn

🔒 OpenVPN server in a Docker container complete with an EasyRSA PKI CA
https://hub.docker.com/r/kylemanna/openvpn/
MIT License
8.72k stars 2.39k forks source link

Having issues when running on kubernetes #536

Open fabn opened 4 years ago

fabn commented 4 years ago

I have a working configuration for this image initially generated with this configuration command:

ovpn_genconfig -u tcp://old.host -z -N -D -s 10.82.171.0/24 -p 'route 192.168.5.0 255.255.255.0

Everything was working on plain docker (actually with rancher1 as orchestrator) and when I connect to vpn from outside I'm able to ping and access any service in 192.168.5.0/24 which is the main purpose of having this VPN.

Now I'm moving everything to kubernetes (plus rancher2) and I migrated old volume to a kubernetes deployment with the same specs as before. I've just changed VPN subnet to be able to have both images running in parallel:

ovpn_genconfig -u tcp://new.host -z -N -D -s 172.16.33.0/24 -p 'route 192.168.5.0 255.255.255.0'

I'd expect everything works as before, however I'm not able to interact with 192.168.5.0/24 network anymore.

I spent a lot of time in debugging this but I wasn't able to make it working. I compared line by line client output and it's identical (except for the networks used), so the issue must be in server configuration.

Here's server log in the old image (working)

15/11/2019 17:24:34RTNETLINK answers: Operation not permitted
15/11/2019 17:24:34RTNETLINK answers: Operation not permitted
15/11/2019 17:24:34RTNETLINK answers: Operation not permitted
15/11/2019 17:24:36Enabling IPv6 Forwarding
15/11/2019 17:24:36sysctl: error setting key 'net.ipv6.conf.default.forwarding': Read-only file system
15/11/2019 17:24:36Failed to enable IPv6 Forwarding default
15/11/2019 17:24:36Failed to enable IPv6 Forwarding
15/11/2019 17:24:36sysctl: error setting key 'net.ipv6.conf.all.forwarding': Read-only file system
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 OpenVPN 2.3.7 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Jul 10 2015
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 library versions: OpenSSL 1.0.2e 3 Dec 2015, LZO 2.09
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Diffie-Hellman initialized with 2048 bit key
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Control Channel Authentication: using '/etc/openvpn/pki/ta.key' as a OpenVPN static key file
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Socket Buffers: R=[87380->131072] S=[16384->131072]
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 ROUTE_GATEWAY 192.168.5.1/255.255.255.0 IFACE=eth0 HWADDR=66:32:30:39:36:37
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 TUN/TAP device tun0 opened
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 TUN/TAP TX queue length set to 100
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 /sbin/ip link set dev tun0 up mtu 1500
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 /sbin/ip addr add dev tun0 local 10.82.171.1 peer 10.82.171.2
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 /sbin/ip route add 192.168.1.0/24 via 10.82.171.2
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 /sbin/ip route add 10.82.171.0/24 via 10.82.171.2
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 GID set to nogroup
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 UID set to nobody
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Listening for incoming TCP connection on [undef]
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 TCPv4_SERVER link local (bound): [undef]
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 TCPv4_SERVER link remote: [undef]
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 MULTI: multi_init called, r=256 v=256
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 IFCONFIG POOL: base=10.82.171.4 size=62, ipv6=0
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 MULTI: TCP INIT maxclients=1024 maxevents=1028
15/11/2019 17:24:36Fri Nov 15 16:24:36 2019 Initialization Sequence Completed

And here's the debug log of the new image

+ set -e 
+ cd /etc/openvpn 
+ USER_ARGS=("${@}") 
+ ARGS=() 
+ addArg --config /etc/openvpn/openvpn.conf 
+ local arg=--config 
+ '[' 2 -ge 1 ']' 
+ local val=/etc/openvpn/openvpn.conf 
+ hasArg --config 
+ local element 
+ return 1 
+ ARGS+=("${arg}") 
+ '[' 2 -ge 1 ']' 
+ ARGS+=("${val}") 
+ source /etc/openvpn/ovpn_env.sh 
++ declare -x OVPN_AUTH= 
++ declare -x OVPN_CIPHER= 
++ declare -x OVPN_CLIENT_TO_CLIENT= 
++ declare -x OVPN_CN=vpn.sportcare.tech 
++ declare -x OVPN_COMP_LZO=1 
++ declare -x OVPN_DEFROUTE=0 
++ declare -x OVPN_DEVICE=tun 
++ declare -x OVPN_DEVICEN=0 
++ declare -x OVPN_DISABLE_PUSH_BLOCK_DNS=0 
++ declare -x OVPN_DNS=0 
++ OVPN_DNS_SERVERS=([0]="8.8.8.8" [1]="8.8.4.4") 
++ declare -x OVPN_DNS_SERVERS 
++ declare -x OVPN_ENV=/etc/openvpn/ovpn_env.sh 
++ OVPN_EXTRA_CLIENT_CONFIG=() 
++ declare -x OVPN_EXTRA_CLIENT_CONFIG 
++ OVPN_EXTRA_SERVER_CONFIG=() 
++ declare -x OVPN_EXTRA_SERVER_CONFIG 
++ declare -x OVPN_FRAGMENT= 
++ declare -x 'OVPN_KEEPALIVE=10 60' 
++ declare -x OVPN_MTU= 
++ declare -x OVPN_NAT=1 
++ declare -x OVPN_PORT=1194 
++ declare -x OVPN_PROTO=tcp 
++ OVPN_PUSH=([0]="route 192.168.5.0 255.255.255.0" [1]="route 192.168.5.0 255.255.255.0") 
++ declare -x OVPN_PUSH 
++ OVPN_ROUTES=([0]="192.168.1.0/24") 
++ declare -x OVPN_ROUTES 
++ declare -x OVPN_SERVER=172.16.33.0/24 
++ declare -x OVPN_SERVER_URL=tcp://vpn.sportcare.tech 
++ declare -x OVPN_TLS_CIPHER= 
+ mkdir -p /dev/net 
+ '[' '!' -c /dev/net/tun ']' 
+ mknod /dev/net/tun c 10 200 
+ '[' -d /etc/openvpn/ccd ']' 
+ addArg --client-config-dir /etc/openvpn/ccd 
+ local arg=--client-config-dir 
+ '[' 2 -ge 1 ']' 
+ local val=/etc/openvpn/ccd 
+ hasArg --client-config-dir 
+ local element 
+ return 1 
+ ARGS+=("${arg}") 
+ '[' 2 -ge 1 ']' 
+ ARGS+=("${val}") 
+ '[' -z '' ']' 
+ OVPN_NATDEVICE=eth0 
+ '[' 0 '!=' 0 ']' 
+ '[' 1 == 1 ']' 
+ setupIptablesAndRouting 
+ iptables -t nat -C POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE 
iptables: No chain/target/match by that name. 
+ iptables -t nat -A POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE 
+ for i in "${OVPN_ROUTES[@]}" 
+ iptables -t nat -C POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE 
iptables: No chain/target/match by that name. 
+ iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE 
+ '[' /etc/openvpn/pki/crl.pem -nt /etc/openvpn/crl.pem ']' 
+ '[' -r /etc/openvpn/crl.pem ']' 
+ addArg --crl-verify /etc/openvpn/crl.pem 
+ local arg=--crl-verify 
+ '[' 2 -ge 1 ']' 
+ local val=/etc/openvpn/crl.pem 
+ hasArg --crl-verify 
+ local element 
+ return 1 
+ ARGS+=("${arg}") 
+ '[' 2 -ge 1 ']' 
+ ARGS+=("${val}") 
+ ip -6 route show default 
+ '[' 0 = 0 ']' 
+ echo 'Enabling IPv6 Forwarding' 
Enabling IPv6 Forwarding 
+ sysctl -w net.ipv6.conf.all.disable_ipv6=0 
sysctl: error setting key 'net.ipv6.conf.all.disable_ipv6': Read-only file system 
+ echo 'Failed to enable IPv6 support' 
Failed to enable IPv6 support 
+ sysctl -w net.ipv6.conf.default.forwarding=1 
sysctl: error setting key 'net.ipv6.conf.default.forwarding': Read-only file system 
+ echo 'Failed to enable IPv6 Forwarding default' 
Failed to enable IPv6 Forwarding default 
+ sysctl -w net.ipv6.conf.all.forwarding=1 
sysctl: error setting key 'net.ipv6.conf.all.forwarding': Read-only file system 
+ echo 'Failed to enable IPv6 Forwarding' 
Failed to enable IPv6 Forwarding 
+ echo 'Running '\''openvpn --config' /etc/openvpn/openvpn.conf --client-config-dir /etc/openvpn/ccd --crl-verify '/etc/openvpn/crl.pem '\''' 
Running 'openvpn --config /etc/openvpn/openvpn.conf --client-config-dir /etc/openvpn/ccd --crl-verify /etc/openvpn/crl.pem ' 
+ exec openvpn --config /etc/openvpn/openvpn.conf --client-config-dir /etc/openvpn/ccd --crl-verify /etc/openvpn/crl.pem 
Fri Nov 15 16:27:14 2019 OpenVPN 2.4.7 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on May  5 2019 
Fri Nov 15 16:27:14 2019 library versions: OpenSSL 1.1.1c  28 May 2019, LZO 2.10 
Fri Nov 15 16:27:14 2019 Diffie-Hellman initialized with 2048 bit key 
Fri Nov 15 16:27:14 2019 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication 
Fri Nov 15 16:27:14 2019 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication 
Fri Nov 15 16:27:14 2019 ROUTE_GATEWAY 169.254.1.1 
Fri Nov 15 16:27:14 2019 TUN/TAP device tun0 opened 
Fri Nov 15 16:27:14 2019 TUN/TAP TX queue length set to 100 
Fri Nov 15 16:27:14 2019 /sbin/ip link set dev tun0 up mtu 1500 
Fri Nov 15 16:27:14 2019 /sbin/ip addr add dev tun0 local 172.16.33.1 peer 172.16.33.2 
Fri Nov 15 16:27:14 2019 /sbin/ip route add 192.168.1.0/24 via 172.16.33.2 
Fri Nov 15 16:27:14 2019 /sbin/ip route add 172.16.33.0/24 via 172.16.33.2 
Fri Nov 15 16:27:14 2019 Could not determine IPv4/IPv6 protocol. Using AF_INET 
Fri Nov 15 16:27:14 2019 Socket Buffers: R=[87380->87380] S=[16384->16384] 
Fri Nov 15 16:27:14 2019 Listening for incoming TCP connection on [AF_INET][undef]:1194 
Fri Nov 15 16:27:14 2019 TCPv4_SERVER link local (bound): [AF_INET][undef]:1194 
Fri Nov 15 16:27:14 2019 TCPv4_SERVER link remote: [AF_UNSPEC] 
Fri Nov 15 16:27:14 2019 GID set to nogroup 
Fri Nov 15 16:27:14 2019 UID set to nobody 
Fri Nov 15 16:27:14 2019 MULTI: multi_init called, r=256 v=256 
Fri Nov 15 16:27:14 2019 IFCONFIG POOL: base=172.16.33.4 size=62, ipv6=0 
Fri Nov 15 16:27:14 2019 MULTI: TCP INIT maxclients=1024 maxevents=1028 
Fri Nov 15 16:27:14 2019 Initialization Sequence Completed

I think the issue should be in this error in the new image

+ iptables -t nat -C POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE 
iptables: No chain/target/match by that name. 
+ iptables -t nat -A POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE 
+ for i in "${OVPN_ROUTES[@]}" 
+ iptables -t nat -C POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE 
iptables: No chain/target/match by that name. 
+ iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE

However if I run iptables after the container is started I can see those rules

+ iptables -t nat -C POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE 
iptables: No chain/target/match by that name. 
+ iptables -t nat -A POSTROUTING -s 172.16.33.0/24 -o eth0 -j MASQUERADE 
+ for i in "${OVPN_ROUTES[@]}" 
+ iptables -t nat -C POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE 
iptables: No chain/target/match by that name. 
+ iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE

Also another strange thing is in this lines:

# Old image
ROUTE_GATEWAY 192.168.5.1/255.255.255.0 IFACE=eth0 HWADDR=66:32:30:39:36:37
# New image
ROUTE_GATEWAY 169.254.1.1

Any hint?

lisenet commented 2 years ago

I know this is a very old issue, but I ran into it as well, and I solved it by creating an init container that sets IP forwarding on the pod.

fabn commented 2 years ago

I know this is a very old issue, but I ran into it as well, and I solved it by creating an init container that sets IP forwarding on the pod.

Interesting, could you please share your setup?

lisenet commented 2 years ago

Sure, I've built an OpenVPN 2.5 Docker image that is running as a deployment pod on Kubernetes 1.22 with a busybox init containter that sets net.ipv4.ip_forward=1. I've a configmap entry that pushes "route 10.11.1.0 255.255.255.0" which is my private subnet. It works both with or without redirect-gateway def1 (the difference being that my client's default GW would change, but I can still access my private subnet either way).