search

pieterlange / kube-openvpn

:closed_lock_with_key: Kubernetes native OpenVPN
MIT License
383 stars 57 forks source link

Can not access to internet, nodes and pods #72

Closed svua closed 4 years ago

svua commented 4 years ago

Hi there,

I have problem with openvpn configuration into k8s. I don't have access to nodes, pods and internet. I have access only to openvpn pod.

My net: one white ip per node 172.16.1.0/24 - node net with 10Gbit link 10.233.0.0/16 - pod net

I installed helm chart with following values:

replicaCount: 1

updateStrategy: {}
  # type: RollingUpdate
  # rollingUpdate:
  #   maxSurge: 1
#   maxUnavailable: 0

# For supporting pulling from private registries
imagePullSecretName:

image:
  repository: jfelten/openvpn-docker
  tag: 1.1.0
  pullPolicy: IfNotPresent
service:
  type: NodePort
  externalPort: 443
  internalPort: 443
  # hostPort: 443
  externalIPs: []
  nodePort: 32443
  # clusterIP: None
  # LoadBalancerSourceRanges: 0.0.0.0/0
  # loadBalancerIP: 10.0.0.1

  ## Here annotations can be added to the openvpn service
  # annotations:
  #   external-dns.alpha.kubernetes.io/hostname: vpn.example.com
  annotations: {}
## Here annotations can be added to the openvpn pod
# podAnnotations:
#   backup.ark.heptio.com/backup-volumes: certs
podAnnotations: {}
# Add privileged init container to enable IPv4 forwarding
ipForwardInitContainer: false

resources:
  limits:
    cpu: 300m
    memory: 256Mi
  requests:
    cpu: 300m
    memory: 256Mi

readinessProbe:
  initialDelaySeconds: 5
  periodSeconds: 5
  successThreshold: 2

persistence:
  enabled: true
  # subPath: openvpn
  ## A manually managed Persistent Volume and Claim
  ## Requires persistence.enabled: true
  ## If defined, PVC must be created manually before volume will be bound
  # existingClaim:

  ## openvpn data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "local-disks"
  accessMode: ReadWriteOnce
  size: 90Mi
openvpn:
  # Network allocated for openvpn clients (default: 10.240.0.0).
  OVPN_NETWORK: 10.240.0.0
  # Network subnet allocated for openvpn client (default: 255.255.0.0).
  OVPN_SUBNET: 255.255.0.0
  # Protocol used by openvpn tcp or udp (default: udp).
  OVPN_PROTO: tcp
  # Kubernetes pod network (optional).
  OVPN_K8S_POD_NETWORK: "10.0.0.0"
  # Kubernetes pod network subnet (optional).
  OVPN_K8S_POD_SUBNET: "255.0.0.0"
  # Kubernetes service network (optional).
  # Define openvpn.OVPN_K8S_SVC_NETWORK and openvpn.OVPN_K8S_SVC_SUBNET if it's needed to create a separate route to Kubernates service subnet
  # OVPN_K8S_SVC_NETWORK:
  # Kubernetes service network subnet (optional).
  # OVPN_K8S_SVC_SUBNET:
  # Set default route which openvpn figures basing on network routes inside openvpn pod
  DEFAULT_ROUTE_ENABLED: true
  # Server certificate data
  # keystoreSecret:
  # secret with openvpn certificates. If specified, certificates are taken from the secret
  # create secret with such command:
  #    kubectl create secret generic openvpn-keystore-secret --from-file=./server.key --from-file=./ca.crt --from-file=./server.crt --from-file=./dh.pem [--from-file=./crl.pem]  [--from-file=./ta.key]
  # Push a `dhcp-option DOMAIN` config
  dhcpOptionDomain: true
  # Redirect all client traffic through VPN
  redirectGateway: true
  # Use/generate certificate revocation list
  useCrl: false
  # Use/generate a ta.key (https://openvpn.net/community-resources/hardening-openvpn-security/)
  taKey: false
  # Override default cipher
  # cipher: AES-256-CBC
  # Lines appended to the end of the server configuration file
  # serverConf: |
  #  max-clients 100
  #  client-to-client
  # Lines appended to the end of the client configuration file
  # Example: if all of your clients are Ubuntu (18.04+) you may need to install
  # the update-systemd-resolved package (apt install update-systemd-resolved) then
  # set the following to make sure systemd-resolved routes DNS requests correctly:
  # clientConf: |
  #  script-security 2
  #  up /etc/openvpn/update-systemd-resolved
  #  up-restart
  #  down /etc/openvpn/update-systemd-resolved
  #  down-pre

  # Enable istio support for openvpn connections
  istio:
    enabled: false
    proxy:
      port: 15001
  iptablesExtra: []
  # - -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
  # - -A FORWARD -m conntrack --ctstate NEW -d 10.240.0.0/255.255.0.0 -j ACCEPT
  # - -A FORWARD -j REJECT

  # Enable CCD support
  ccd:
    enabled: false
    config: {}
    # johndoe: "ifconfig-push 10.240.100.10 10.240.100.11"
    # janedoe: "ifconfig-push 10.240.100.20 10.240.100.21"

nodeSelector:
  kubernetes.io/hostname: ch1
tolerations: []

My key

client
nobind
dev tun
remote *.*.*.* 32443 tcp

redirect-gateway def1

<key>
-----BEGIN PRIVATE KEY-----
...

Now I connect to vpn

# openvpn --config kubeVPN.ovpn
Wed Jul  8 19:28:20 2020 OpenVPN 2.4.6 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on May 14 2019
Wed Jul  8 19:28:20 2020 library versions: OpenSSL 1.1.1  11 Sep 2018, LZO 2.10
Wed Jul  8 19:28:20 2020 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Wed Jul  8 19:28:20 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]*.*.*.*:32443
Wed Jul  8 19:28:20 2020 Attempting to establish TCP connection with [AF_INET]*.*.*.*:32443 [nonblock]
Wed Jul  8 19:28:21 2020 TCP connection established with [AF_INET]*.*.*.*:32443
Wed Jul  8 19:28:21 2020 TCP_CLIENT link local: (not bound)
Wed Jul  8 19:28:21 2020 TCP_CLIENT link remote: [AF_INET]*.*.*.*:32443
Wed Jul  8 19:28:21 2020 WARNING: 'keydir' is present in remote config but missing in local config, remote='keydir 0'
Wed Jul  8 19:28:21 2020 [server] Peer Connection Initiated with [AF_INET]*.*.*.*:32443
Wed Jul  8 19:28:23 2020 WARNING: INSECURE cipher with block size less than 128 bit (64 bit).  This allows attacks like SWEET32.  Mitigate by using a --cipher with a larger block size (e.g. AES-256-CBC).
Wed Jul  8 19:28:23 2020 WARNING: INSECURE cipher with block size less than 128 bit (64 bit).  This allows attacks like SWEET32.  Mitigate by using a --cipher with a larger block size (e.g. AES-256-CBC).
Wed Jul  8 19:28:23 2020 WARNING: cipher with small block size in use, reducing reneg-bytes to 64MB to mitigate SWEET32 attacks.
Wed Jul  8 19:28:23 2020 TUN/TAP device tun0 opened
Wed Jul  8 19:28:23 2020 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Wed Jul  8 19:28:23 2020 /sbin/ip link set dev tun0 up mtu 1500
Wed Jul  8 19:28:23 2020 /sbin/ip addr add dev tun0 local 10.240.0.6 peer 10.240.0.5
Wed Jul  8 19:28:23 2020 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Wed Jul  8 19:28:23 2020 Initialization Sequence Completed

My routers

# ip r
0.0.0.0/1 via 10.240.0.5 dev tun0
default via 192.168.1.1 dev enp8s0 proto dhcp metric 20100
10.0.0.0/8 via 10.240.0.5 dev tun0
10.233.84.42 via 10.240.0.5 dev tun0
10.240.0.1 via 10.240.0.5 dev tun0
10.240.0.5 dev tun0 proto kernel scope link src 10.240.0.6
*.*.*.* via 192.168.1.1 dev enp8s0
*.*.*.* via 192.168.1.1 dev enp8s0 proto static metric 100
116.202.15.89 via 192.168.1.1 dev enp8s0 proto static metric 100
128.0.0.0/1 via 10.240.0.5 dev tun0
169.254.0.0/16 dev virbr5 scope link metric 1000 linkdown
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-081d3729f08e proto kernel scope link src 172.18.0.1 linkdown
192.168.1.0/24 dev enp8s0 proto kernel scope link src 192.168.1.111 metric 100
192.168.1.1 dev enp8s0 proto static scope link metric 100
192.168.39.0/24 dev virbr6 proto kernel scope link src 192.168.39.1 linkdown
192.168.101.0/24 dev virbr1 proto kernel scope link src 192.168.101.1 linkdown
192.168.102.0/24 dev virbr2 proto kernel scope link src 192.168.102.1 linkdown
192.168.103.0/24 dev virbr3 proto kernel scope link src 192.168.103.1 linkdown
192.168.104.0/24 dev virbr4 proto kernel scope link src 192.168.104.1 linkdown
192.168.105.0/24 dev virbr5 proto kernel scope link src 192.168.105.1 linkdown
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

And pings

# ping 10.240.0.5
PING 10.240.0.5 (10.240.0.5) 56(84) bytes of data.
^C
--- 10.240.0.5 ping statistics ---
20 packets transmitted, 0 received, 100% packet loss, time 483ms

# ping 10.240.0.1
PING 10.240.0.1 (10.240.0.1) 56(84) bytes of data.
64 bytes from 10.240.0.1: icmp_seq=1 ttl=64 time=34.9 ms
64 bytes from 10.240.0.1: icmp_seq=2 ttl=64 time=35.2 ms
64 bytes from 10.240.0.1: icmp_seq=3 ttl=64 time=34.8 ms
^C
--- 10.240.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 5ms
rtt min/avg/max/mdev = 34.799/34.945/35.167/0.159 ms

# ping 10.233.84.42
PING 10.233.84.42 (10.233.84.42) 56(84) bytes of data.
64 bytes from 10.233.84.42: icmp_seq=1 ttl=64 time=37.5 ms
64 bytes from 10.233.84.42: icmp_seq=2 ttl=64 time=37.6 ms
64 bytes from 10.233.84.42: icmp_seq=3 ttl=64 time=50.6 ms
^C
--- 10.233.84.42 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 6ms
rtt min/avg/max/mdev = 37.544/41.924/50.606/6.143 ms

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
19 packets transmitted, 0 received, 100% packet loss, time 448ms

Why do not I have access to 8.8.8.8? Also I don't have access to pod network 10.233.0.0/16. I have access only to openvpn pod with ip 10.233.84.42.

thekudryash commented 3 years ago

@svua Hello! Did you resolve this issue ?

svua commented 3 years ago

Did you resolve this issue ?

I specified externalIPs and after that it worked for me

service:
  type: NodePort
  # externalPort: 443
  # internalPort: 443
  # hostPort: 443
  externalIPs:
    - XX.XX.XX.XX
  nodePort: 30443

externalIPs meen main ip on the server with default router to world