search

helm / charts

⚠️(OBSOLETE) Curated applications for Kubernetes
Apache License 2.0
15.49k stars 16.81k forks source link

stable/openVpn Connection reset, restarting [0] SIGUSR1[soft,connection-reset] received, client-instance restarting #16603

Closed getoutout closed 5 years ago

getoutout commented 5 years ago

request for help

[root@master ~]# helm list NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
invincible-mite 1 Fri Aug 23 18:27:30 2019 DEPLOYED nginx-ingress-1.8.1 0.24.1 default
nginx-ingress 5 Sun Aug 25 17:10:47 2019 DEPLOYED nginx-ingress-1.8.1 0.24.1 ingress-nginx openvpn 1 Sun Aug 25 20:46:56 2019 DEPLOYED openvpn-3.13.8 1.1.0 kube-system
[root@master ~]# helm version Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"} [root@master ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

What happened: client log: Sun Aug 25 13:28:00 2019 127.0.0.1:35876 SIGUSR1[soft,connection-reset] received, client-instance restarting Sun Aug 25 13:28:05 2019 TCP connection established with [AF_INET]127.0.0.1:44268 Sun Aug 25 13:28:05 2019 127.0.0.1:44268 Connection reset, restarting [0] Sun Aug 25 13:28:05 2019 127.0.0.1:44268 SIGUSR1[soft,connection-reset] received, client-instance restarting Sun Aug 25 13:28:10 2019 TCP connection established with [AF_INET]127.0.0.1:41946 Sun Aug 25 13:28:10 2019 127.0.0.1:41946 Connection reset, restarting [0] Sun Aug 25 13:28:10 2019 127.0.0.1:41946 SIGUSR1[soft,connection-reset] received, client-instance restarting Sun Aug 25 13:28:15 2019 TCP connection established with [AF_INET]127.0.0.1:42429 Sun Aug 25 13:28:15 2019 127.0.0.1:42429 Connection reset, restarting [0] Sun Aug 25 13:28:15 2019 127.0.0.1:42429 SIGUSR1[soft,connection-reset] received, client-instance restarting Sun Aug 25 13:28:20 2019 TCP connection established with [AF_INET]127.0.0.1:39077 Sun Aug 25 13:28:20 2019 127.0.0.1:39077 Connection reset, restarting [0] Sun Aug 25 13:28:20 2019 127.0.0.1:39077 SIGUSR1[soft,connection-reset] received, client-instance restarting Sun Aug 25 13:28:25 2019 TCP connection established with [AF_INET]127.0.0.1:43794 Sun Aug 25 13:28:25 2019 127.0.0.1:43794 Connection reset, restarting [0] Sun Aug 25 13:28:25 2019 127.0.0.1:43794 SIGUSR1[soft,connection-reset] received, client-instance restarting Sun Aug 25 13:28:30 2019 TCP connection established with [AF_INET]127.0.0.1:42780 Sun Aug 25 13:28:30 2019 127.0.0.1:42780 Connection reset, restarting [0] Sun Aug 25 13:28:30 2019 127.0.0.1:42780 SIGUSR1[soft,connection-reset] received, client-instance restarting Sun Aug 25 13:28:35 2019 TCP connection established with [AF_INET]127.0.0.1:46862 Sun Aug 25 13:28:35 2019 127.0.0.1:46862 Connection reset, restarting [0] Sun Aug 25 13:28:35 2019 127.0.0.1:46862 SIGUSR1[soft,connection-reset] received, client-instance restarting Sun Aug 25 13:28:40 2019 TCP connection established with [AF_INET]127.0.0.1:38739 Sun Aug 25 13:28:40 2019 127.0.0.1:38739 Connection reset, restarting [0] Sun Aug 25 13:28:40 2019 127.0.0.1:38739 SIGUSR1[soft,connection-reset] received, client-instance restarting Sun Aug 25 13:28:45 2019 TCP connection established with [AF_INET]127.0.0.1:35464 Sun Aug 25 13:28:45 2019 127.0.0.1:35464 Connection reset, restarting [0] Sun Aug 25 13:28:45 2019 127.0.0.1:35464 SIGUSR1[soft,connection-reset] received, client-instance restarting Sun Aug 25 13:28:50 2019 TCP connection established with [AF_INET]127.0.0.1:41096 Sun Aug 25 13:28:50 2019 127.0.0.1:41096 Connection reset, restarting [0] Sun Aug 25 13:28:50 2019 127.0.0.1:41096 SIGUSR1[soft,connection-reset] received, client-instance restarting Sun Aug 25 13:28:55 2019 TCP connection established with [AF_INET]127.0.0.1:39045 Sun Aug 25 13:28:55 2019 127.0.0.1:39045 Connection reset, restarting [0] Sun Aug 25 13:28:55 2019 127.0.0.1:39045 SIGUSR1[soft,connection-reset] received, client-instance restarting ......

MarcoAbi commented 5 years ago

I'm having the same issue, I'm using this chart inside Rancher 2. Do you have some clue on this behavior?

kaskavalci commented 5 years ago

I solved it by enabling UDP protocol. 

$ helm install --set openvpn.OVPN_PROTO=udp stable/openvpn
getoutout commented 5 years ago

I solved it by enabling UDP protocol.

$ helm install --set openvpn.OVPN_PROTO=udp stable/openvpn

thank you , use udp can connect to openvpn server,but can't resolve kubernetes dns,can't ping kubernetes pod ip

openvpn server logs Fri Oct 4 14:23:54 2019 100.85.170.130:63496 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA Fri Oct 4 14:23:54 2019 100.85.170.130:63496 [client] Peer Connection Initiated with [AF_INET]100.85.170.130:63496 Fri Oct 4 14:23:54 2019 client/100.85.170.130:63496 MULTI_sva: pool returned IPv4=10.240.0.6, IPv6=(Not enabled) Fri Oct 4 14:23:54 2019 client/100.85.170.130:63496 MULTI: Learn: 10.240.0.6 -> client/100.85.170.130:63496 Fri Oct 4 14:23:54 2019 client/100.85.170.130:63496 MULTI: primary virtual IP for client/100.85.170.130:63496: 10.240.0.6 Fri Oct 4 14:23:55 2019 client/100.85.170.130:63496 PUSH: Received control message: 'PUSH_REQUEST' Fri Oct 4 14:23:55 2019 client/100.85.170.130:63496 send_push_reply(): safe_cap=940 Fri Oct 4 14:23:55 2019 client/100.85.170.130:63496 SENT CONTROL [client]: 'PUSH_REPLY,route 100.84.235.5 255.255.255.255,route 10.244.0.0 255.255.0.0,route 10.96.0.0 255.240.0.0,dhcp-option DOMAIN-SEARCH kube-system.svc.cluster.local,dhcp-option DOMAIN-SEARCH svc.cluster.local,dhcp-option DOMAIN-SEARCH cluster.local,dhcp-option DNS 10.96.0.10,route 10.240.0.1,topology net30,ping 10,ping-restart 60,ifconfig 10.240.0.6 10.240.0.5' (status=1) Fri Oct 4 14:28:09 2019 100.85.170.130:59383 TLS: Initial packet from [AF_INET]100.85.170.130:59383, sid=95a6619f 61ae058d Fri Oct 4 14:28:09 2019 100.85.170.130:59383 VERIFY OK: depth=1, CN=ca\n Fri Oct 4 14:28:09 2019 100.85.170.130:59383 VERIFY OK: depth=0, CN=client Fri Oct 4 14:28:09 2019 100.85.170.130:59383 WARNING: 'keydir' is present in local config but missing in remote config, local='keydir 1' Fri Oct 4 14:28:09 2019 100.85.170.130:59383 Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key Fri Oct 4 14:28:09 2019 100.85.170.130:59383 WARNING: INSECURE cipher with block size less than 128 bit (64 bit). This allows attacks like SWEET32. Mitigate by using a --cipher with a larger block size (e.g. AES-256-CBC). Fri Oct 4 14:28:09 2019 100.85.170.130:59383 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication Fri Oct 4 14:28:09 2019 100.85.170.130:59383 Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key Fri Oct 4 14:28:09 2019 100.85.170.130:59383 WARNING: INSECURE cipher with block size less than 128 bit (64 bit). This allows attacks like SWEET32. Mitigate by using a --cipher with a larger block size (e.g. AES-256-CBC). Fri Oct 4 14:28:09 2019 100.85.170.130:59383 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication Fri Oct 4 14:28:09 2019 100.85.170.130:59383 WARNING: cipher with small block size in use, reducing reneg-bytes to 64MB to mitigate SWEET32 attacks. Fri Oct 4 14:28:09 2019 100.85.170.130:59383 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA Fri Oct 4 14:28:09 2019 100.85.170.130:59383 [client] Peer Connection Initiated with [AF_INET]100.85.170.130:59383 Fri Oct 4 14:28:09 2019 MULTI: new connection by client 'client' will cause previous active sessions by this client to be dropped. Remember to use the --duplicate-cn option if you want multiple clients using the same certificate or username to concurrently connect. Fri Oct 4 14:28:09 2019 MULTI_sva: pool returned IPv4=10.240.0.6, IPv6=(Not enabled) Fri Oct 4 14:28:09 2019 MULTI: Learn: 10.240.0.6 -> client/100.85.170.130:59383 Fri Oct 4 14:28:09 2019 MULTI: primary virtual IP for client/100.85.170.130:59383: 10.240.0.6 Fri Oct 4 14:28:11 2019 client/100.85.170.130:59383 PUSH: Received control message: 'PUSH_REQUEST' Fri Oct 4 14:28:11 2019 client/100.85.170.130:59383 send_push_reply(): safe_cap=940 Fri Oct 4 14:28:11 2019 client/100.85.170.130:59383 SENT CONTROL [client]: 'PUSH_REPLY,route 100.84.235.5 255.255.255.255,route 10.244.0.0 255.255.0.0,route 10.96.0.0 255.240.0.0,dhcp-option DOMAIN-SEARCH kube-system.svc.cluster.local,dhcp-option DOMAIN-SEARCH svc.cluster.local,dhcp-option DOMAIN-SEARCH cluster.local,dhcp-option DNS 10.96.0.10,route 10.240.0.1,topology net30,ping 10,ping-restart 60,ifconfig 10.240.0.6 10.240.0.5' (status=1)

opnvpn client logs

Fri Oct 04 21:47:26 2019 SIGHUP[hard,] received, process restarting Fri Oct 04 21:47:26 2019 OpenVPN 2.4.6 x86_64-w64-mingw32 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] built on Apr 24 2018 Fri Oct 04 21:47:26 2019 Windows version 6.1 (Windows 7) 64bit Fri Oct 04 21:47:26 2019 library versions: OpenSSL 1.1.0h 27 Mar 2018, LZO 2.10 Fri Oct 04 21:47:31 2019 WARNING: No server certificate verification method has been enabled. See http://openvpn.net/howto.html#mitm for more info. Fri Oct 04 21:47:31 2019 TCP/UDP: Preserving recently used remote address: [AF_INET]192.168.4.191:32443 Fri Oct 04 21:47:31 2019 UDP link local: (not bound) Fri Oct 04 21:47:31 2019 UDP link remote: [AF_INET]192.168.4.191:32443 Fri Oct 04 21:47:31 2019 WARNING: 'keydir' is present in remote config but missing in local config, remote='keydir 0' Fri Oct 04 21:47:31 2019 [server] Peer Connection Initiated with [AF_INET]192.168.4.191:32443 Fri Oct 04 21:47:32 2019 Options error: --dhcp-option: unknown option type 'DOMAIN-SEARCH' or missing or unknown parameter Fri Oct 04 21:47:32 2019 Options error: --dhcp-option: unknown option type 'DOMAIN-SEARCH' or missing or unknown parameter Fri Oct 04 21:47:32 2019 Options error: --dhcp-option: unknown option type 'DOMAIN-SEARCH' or missing or unknown parameter Fri Oct 04 21:47:32 2019 WARNING: INSECURE cipher with block size less than 128 bit (64 bit). This allows attacks like SWEET32. Mitigate by using a --cipher with a larger block size (e.g. AES-256-CBC). Fri Oct 04 21:47:32 2019 WARNING: INSECURE cipher with block size less than 128 bit (64 bit). This allows attacks like SWEET32. Mitigate by using a --cipher with a larger block size (e.g. AES-256-CBC). Fri Oct 04 21:47:32 2019 WARNING: cipher with small block size in use, reducing reneg-bytes to 64MB to mitigate SWEET32 attacks. Fri Oct 04 21:47:32 2019 open_tun Fri Oct 04 21:47:32 2019 TAP-WIN32 device [本地连接 3] opened: \.\Global{3290C129-C6C8-41AC-A217-0E2404A65CFD}.tap Fri Oct 04 21:47:32 2019 Notified TAP-Windows driver to set a DHCP IP/netmask of 10.240.0.6/255.255.255.252 on interface {3290C129-C6C8-41AC-A217-0E2404A65CFD} [DHCP-serv: 10.240.0.5, lease-time: 31536000] Fri Oct 04 21:47:32 2019 Successful ARP Flush on interface [21] {3290C129-C6C8-41AC-A217-0E2404A65CFD} Fri Oct 04 21:47:32 2019 do_ifconfig, tt->did_ifconfig_ipv6_setup=0 Fri Oct 04 21:47:37 2019 Initialization Sequence Completed

C:\Users\admin>nslookup kubernetes.default.svc.cluster.local DNS request timed out. timeout was 2 seconds. 服务器: UnKnown Address: 10.96.0.10

DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** 请求 UnKnown 超时

guysoft commented 4 years ago

I solved it by enabling UDP protocol.

$ helm install --set openvpn.OVPN_PROTO=udp stable/openvpn

This solution does not work on AWS due to services not supporting UDP. EKS does not get it. https://github.com/kubernetes/kubernetes/issues/79523

ghost commented 4 years ago

I am also facing the same issue , connection reset problem is resolved by switching to UDP protocol but not able to connect to services on ClusterIP not did on pod ip's , Just for information i am working on a GKE private cluster

Martin-Hogge commented 3 years ago

In my case when I'm switching to udp I get these logs:

Info 2020-11-23 11:37:35.489 CET Mon Nov 23 10:37:35 2020 /sbin/ip route del 10.240.0.0/16
Error 2020-11-23 11:37:35.489 CET RTNETLINK answers: Operation not permitted
Info 2020-11-23 11:37:35.489 CET Mon Nov 23 10:37:35 2020 ERROR: Linux route delete command failed: external program exited with error status: 2
Info 2020-11-23 11:37:35.489 CET Mon Nov 23 10:37:35 2020 Closing TUN/TAP interface
Info 2020-11-23 11:37:35.489 CET Mon Nov 23 10:37:35 2020 /sbin/ip addr del dev tun0 local 10.240.0.1 peer 10.240.0.2
Info 2020-11-23 11:37:35.489 CET Mon Nov 23 10:37:35 2020 Linux ip addr del failed: external program exited with error status: 2
Error 2020-11-23 11:37:35.489 CET RTNETLINK answers: Operation not permitted
Info 2020-11-23 11:37:35.501 CET Mon Nov 23 10:37:35 2020 SIGTERM[hard,] received, process exiting

If I keep tcp I'am able to open a session but not able to call clusterIp of my pods (connection timeout).

InsOpDe commented 3 years ago

The initial issue is caused by the readiness check nc -z 127.0.0.1 443 which has a 5 second interval, thus the log gets created each 5 seconds