search

submariner-io / submariner

Networking component for interconnecting Pods and Services across Kubernetes clusters.
https://submariner.io
Apache License 2.0
2.42k stars 190 forks source link

strongswan status exit code 3, even when "security associations" is up, duplicate SAs #360

Closed manosnoam closed 4 years ago

manosnoam commented 4 years ago 
[nmanos@nmanos ocp-install]$ oc exec submariner-gateway-d4t87 bash -ti -n submariner-operator 
strongswan statusall

Status of IKE charon daemon (strongSwan 5.8.2, Linux 4.18.0-147.3.1.el8_1.x86_64, x86_64):
  uptime: 17 hours, since Feb 19 16:24:07 2020
  malloc: sbrk 3313664, mmap 0, used 2204688, free 1108976
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 4
  loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm drbg curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
Listening IP addresses:
  10.0.59.17
  10.130.2.1
  240.0.59.17
Connections:
submariner-cable-subm-cluster-b-10-166-0-31:  10.0.59.17...66.187.233.202  IKEv2
submariner-cable-subm-cluster-b-10-166-0-31:   local:  [107.23.91.226] uses pre-shared key authentication
submariner-cable-subm-cluster-b-10-166-0-31:   remote: [66.187.233.202] uses pre-shared key authentication
submariner-child-submariner-cable-subm-cluster-b-10-166-0-31:   child:  10.0.59.17/32 172.30.0.0/16 10.128.0.0/14 === 10.166.0.31/32 100.96.0.0/16 10.252.0.0/14 TUNNEL
Security Associations (1 up, 1 connecting):
submariner-cable-subm-cluster-b-10-166-0-31[6]: ESTABLISHED 2 hours ago, 10.0.59.17[107.23.91.226]...66.187.233.202[66.187.233.202]
submariner-cable-subm-cluster-b-10-166-0-31[6]: IKEv2 SPIs: 47f75baf8a0b2fce_i* c29a132dc3778a4d_r, rekeying in 91 minutes
submariner-cable-subm-cluster-b-10-166-0-31[6]: IKE proposal: AES_GCM_16_128/PRF_HMAC_SHA2_256/MODP_2048
submariner-child-submariner-cable-subm-cluster-b-10-166-0-31{19}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: cced9dba_i cc0777f9_o
submariner-child-submariner-cable-subm-cluster-b-10-166-0-31{19}:  AES_GCM_16_128/MODP_2048, 0 bytes_i, 0 bytes_o, rekeying in 20 minutes
submariner-child-submariner-cable-subm-cluster-b-10-166-0-31{19}:   10.0.59.17/32 10.128.0.0/14 172.30.0.0/16 === 10.166.0.31/32 10.252.0.0/14 100.96.0.0/16
submariner-cable-subm-cluster-b-10-166-0-31[1]: CONNECTING, 10.0.59.17[%any]...66.187.233.202[%any]
submariner-cable-subm-cluster-b-10-166-0-31[1]: IKEv2 SPIs: 4fb26e3da8589e82_i* 0000000000000000_r
submariner-cable-subm-cluster-b-10-166-0-31[1]: Tasks active: IKE_VENDOR IKE_INIT IKE_NATD IKE_CERT_PRE IKE_AUTH IKE_CERT_POST IKE_CONFIG CHILD_CREATE IKE_AUTH_LIFETIME 

[root@ip-10-0-59-17 submariner]# echo $?
3
mangelajo commented 4 years ago

Probably the "CONNECTING" part, with a repeated connection is what we need to investigate. I see those kinds of duplicate connection "SAs" sometimes.

sridhargaddam commented 4 years ago

@manosnoam, we use the VICI interface to control Charon in Submariner Gateway. Its recommended using swanctl, a command-line tool, to configure and control Charon.

So, please use the following command.

swanctl --list-sas (or swanctl --list-conn)

Sample output:

[root@cluster2-worker submariner]# swanctl --list-sas submariner-cable-cluster3-172-17-0-7: #2, ESTABLISHED, IKEv2, e1d379930d5dda7d_i 7fa34a6224864124_r* local '172.17.0.10' @ 172.17.0.10[4500] remote '172.17.0.7' @ 172.17.0.7[4500] AES_GCM_16-128/PRF_HMAC_SHA2_256/MODP_2048 established 1379s ago, rekeying in 13013s submariner-child-submariner-cable-cluster3-172-17-0-7: #1, reqid 1, INSTALLED, TUNNEL-in-UDP, ESP:AES_GCM_16-128 installed 1379s ago, rekeying in 1883s, expires in 2581s in cac21d13, 0 bytes, 0 packets out cb66f8b2, 0 bytes, 0 packets local 169.254.2.0/24 172.17.0.10/32 remote 169.254.3.0/24 172.17.0.7/32 [root@cluster2-worker submariner]# [root@cluster2-worker submariner]# echo $? 0 [root@cluster2-worker submariner]#

Alternately, this would also work, but not recommended (AFAIU)

[root@cluster2-worker submariner]# strongswan stroke statusall Status of IKE charon daemon (strongSwan 5.7.2, Linux 5.5.5-200.fc31.x86_64, x86_64): uptime: 23 minutes, since Feb 25 04:33:50 2020 malloc: sbrk 2965504, mmap 0, used 1128352, free 1837152 worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 2 loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters Listening IP addresses: 10.245.128.0 172.17.0.10 240.17.0.10 Connections: submariner-cable-cluster3-172-17-0-7: 172.17.0.10...172.17.0.7 IKEv2 submariner-cable-cluster3-172-17-0-7: local: [172.17.0.10] uses pre-shared key authentication submariner-cable-cluster3-172-17-0-7: remote: [172.17.0.7] uses pre-shared key authentication submariner-child-submariner-cable-cluster3-172-17-0-7: child: 172.17.0.10/32 169.254.2.0/24 === 172.17.0.7/32 169.254.3.0/24 TUNNEL Security Associations (1 up, 0 connecting): submariner-cable-cluster3-172-17-0-7[2]: ESTABLISHED 23 minutes ago, 172.17.0.10[172.17.0.10]...172.17.0.7[172.17.0.7] submariner-cable-cluster3-172-17-0-7[2]: IKEv2 SPIs: e1d379930d5dda7d_i 7fa34a6224864124_r*, rekeying in 3 hours submariner-cable-cluster3-172-17-0-7[2]: IKE proposal: AES_GCM_16_128/PRF_HMAC_SHA2_256/MODP_2048 submariner-child-submariner-cable-cluster3-172-17-0-7{1}: INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: cac21d13_i cb66f8b2_o submariner-child-submariner-cable-cluster3-172-17-0-7{1}: AES_GCM_16_128, 0 bytes_i, 0 bytes_o, rekeying in 30 minutes submariner-child-submariner-cable-cluster3-172-17-0-7{1}: 169.254.2.0/24 172.17.0.10/32 === 169.254.3.0/24 172.17.0.7/32 [root@cluster2-worker submariner]# [root@cluster2-worker submariner]# echo $? 0 [root@cluster2-worker submariner]#

manosnoam commented 4 years ago

Thanks Sridhar, I will try those commands. But maybe exit code 3 is an indication of a bug due to: Security Associations (1 up, 1 connecting) Which seems like an attempt to create a second connection, while it should/can not, cause only one connection is required?

sridhargaddam commented 4 years ago

Thanks Sridhar, I will try those commands. But maybe exit code 3 is an indication of a bug due to: Security Associations (1 up, 1 connecting) Which seems like an attempt to create a second connection, while it should/can not, cause only one connection is required?

I checked locally in my cluster and even when the connection is successfully established, I see that "strongswan statusall" returns exit code of 3. You can give a try with the commands I suggested and let us know if you still see exit code of 3.

sridhargaddam commented 4 years ago

@manosnoam as we now have an API to query the status of IPsec tunnels in a generic manner (irrespective of the cableEngine that is used in the cluster), please continue to use the new API over these commands. I'm closing this issue.

kubectl describe Gateways -n submariner-operator

ygm521 commented 2 years ago

how to know kernel support AES_GCM_16_128? thanks