Central controller doesn't establish ip tunnel

[rreisim]

Hi, this is my second issue and hopefully the first to get an answer.

Once I have successfully installed the ICN SD-EWAN solution on two different virtual machines, I am trying to perform a use case on one machine acting as a hub and the other acting as an egde node. To implement the following scheme I have used the central controller solution.

IP hub - 192.168.250.185 IP edge - 192.168.250.156

First, i created an overlay.

Next, I made the proposal aggregation in the overlay.

Following this structure, I made the iprange to serve ips to the edge nodes. (Subnet 192.168.10.0/24)

Finally, i registrered the hub with kubeconfig encode in base64.

And the edge node too.

Ok, so far so good. The important issue is when I created the hub-device connection. It seems that the edge node has a virtual ip "192.168.10.1".

The edge status shows the success of the process.

But when I look at the subnets in the edge node cnf... surprise! The subnet doesnt not exists.

Can you explain me what is the error I am making or if this software has some bugs?

You can contact with me at rreisim@upv.es

I hope you will give me an answer as soon as possible. Best regards. Raúl.

[ravaga]

I'm also interested in this issue because I also tried to run a similar use case and I don't achieved the expected results, the configured subnets are not actually created on the edge node.

[Airren]

net2 is used as the underlay network for the IPsec tunnel. Please check the connectivity through net2 between Hub CNF and Edge CNF first.

[Ruoyu-y]

Hi, this is my second issue and hopefully the first to get an answer.

Once I have successfully installed the ICN SD-EWAN solution on two different virtual machines, I am trying to perform a use case on one machine acting as a hub and the other acting as an egde node. To implement the following scheme I have used the central controller solution.

IP hub - 192.168.250.185 IP edge - 192.168.250.156

First, i created an overlay.

Next, I made the proposal aggregation in the overlay.

Following this structure, I made the iprange to serve ips to the edge nodes. (Subnet 192.168.10.0/24)

Finally, i registrered the hub with kubeconfig encode in base64.

And the edge node too.

Ok, so far so good. The important issue is when I created the hub-device connection. It seems that the edge node has a virtual ip "192.168.10.1".

The edge status shows the success of the process.

But when I look at the subnets in the edge node cnf... surprise! The subnet doesnt not exists.

Can you explain me what is the error I am making or if this software has some bugs?

You can contact with me at rreisim@upv.es

I hope you will give me an answer as soon as possible. Best regards. Raúl.

Hi Raúl, I am wondering if these two public ip address IP hub - 192.168.250.185 IP edge - 192.168.250.156 are the public ip address of your vm. Our solution attempt to establish an ipsec tunnel between the SDEWAN CNFs reside on your machine, so the public ip address used during hub/edge registration shall be the one within the cnf, the ip address of net2 interface as @Airren mentioned. So please, check the network connection between the two SDEWAN CNFs and re-register with the correct ips.

[rreisim]

net2 is used as the underlay network for the IPsec tunnel. Please check the connectivity through net2 between Hub CNF and Edge CNF first.

Hi Raúl, I am wondering if these two public ip address IP hub - 192.168.250.185 IP edge - 192.168.250.156 are the public ip address of your vm. Our solution attempt to establish an ipsec tunnel between the SDEWAN CNFs reside on your machine, so the public ip address used during hub/edge registration shall be the one within the cnf, the ip address of net2 interface as @Airren mentioned. So please, check the network connection between the two SDEWAN CNFs and re-register with the correct ips.

Hi @Ruoyu-y and @Airren, thank you so much for your answers.

In fact, I didn't have connection between CNF through net2.

Now, the schema is this:

Hub - 192.168.250.174 (PIP) - 10.10.70.40 (net2)

Edge1 - 192.168.250.156 (PIP) - 10.10.70.41 (net2)

The connection between both CNFs is now available, so I can ping it successfully.

I also re-registered the hub and edge1 with their respective acquired IPs (I had problems with registering both deployments using only the CNF IP, so I added both the PIP and the one generated by net2 on the CNF).

Once the connection between the hub and edge1 has been created, the crds proposal and IpsecHost (in the case of the edge) and IpsecSite (in the case of the Hub) are successfully created and appear to have successful configurations.

HUB

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: IpsecSite
metadata:
  creationTimestamp: "2022-11-17T12:52:20Z"
  finalizers:
  - ipsec.site.finalizers.sdewan.akraino.org
  generation: 1
  labels:
    emco/deployment-id: 3720661577566571836-hub1-app
    sdewanPurpose: base
    targetCluster: Hub.hub1
  name: hub1device2
  namespace: default
  resourceVersion: "28795"
  uid: e1364ee2-2a1e-40e0-911b-04873784ec83
spec:
  authentication_method: pubkey
  connections:
  - conn_type: tunnel
    crypto_proposal:
    - proposal1
    local_subnet: 0.0.0.0/0
    local_updown: /etc/updown
    mark: "30"
    mode: start
    name: Conndevice2_192168101
    remote_sourceip: 192.168.10.1
  crypto_proposal:
  - proposal1
  force_crypto_proposal: "0"
  local_identifier: CN=hub-hub1-cert
  local_private_cert: <local_private_cert>
  local_public_cert: <local_public_cert>
  remote: '%any'
  remote_identifier: CN=device-device2-cert
  shared_ca: <shared_ca>
  type: VTI-based
status:
  appliedGeneration: 1
  appliedTime: "2022-11-17T12:52:21Z"
  state: In Sync

EDGE1

apiVersion: v1
items:
- apiVersion: batch.sdewan.akraino.org/v1alpha1
  kind: IpsecHost
  metadata:
    creationTimestamp: "2022-11-17T12:52:21Z"
    finalizers:
    - ipsec.host.finalizers.sdewan.akraino.org
    generation: 2
    labels:
      emco/deployment-id: 8347335641058164639-device2-app
      sdewanPurpose: base
      targetCluster: Device.device2
    name: device2hub1
    namespace: default
    resourceVersion: "288456"
    uid: e771a9c2-f62a-420d-bff3-1c73a9c7edfc
  spec:
    authentication_method: pubkey
    connections:
    - conn_type: tunnel
      crypto_proposal:
      - proposal1
      local_sourceip: '%config'
      local_updown: /etc/updown_oip
      mode: start
      name: Connhub1_192168250174
      remote_subnet: 0.0.0.0/0
    crypto_proposal:
    - proposal1
    force_crypto_proposal: "0"
    local_identifier: CN=device-device2-cert
    local_private_cert: <local_private_cert>
    remote: 192.168.250.174
    remote_identifier: CN=hub-hub1-cert
    shared_ca: <shared_ca>
    type: policy-based
  status:
    appliedGeneration: 2
    appliedTime: "2022-11-17T12:52:22Z"
    state: In Sync
- apiVersion: batch.sdewan.akraino.org/v1alpha1
  kind: IpsecHost
  metadata:
    annotations:
    creationTimestamp: "2022-11-17T12:03:52Z"
    generation: 1
    labels:
      sdewanPurpose: device2
    name: ipsechost
    namespace: default
    resourceVersion: "278299"
    uid: 3b01abba-95a1-43f7-9fe1-c56648b18340
  spec:
    authentication_method: psk
    connections:
    - conn_type: tunnel
      crypto_proposal:
      - ipsecproposal
      local_sourceip: 192.168.250.156
      mode: start
      name: connA
      remote_subnet: 192.168.1.1/24,$10.10.70.40/32
    crypto_proposal:
    - ipsecproposal
    force_crypto_proposal: "0"
    local_identifier: 10.10.70.41
    name: edgeA
    pre_shared_key: test_key
    remote: 10.10.70.40
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Despite all this, edge1 still does not add the new network interface correctly.

At this point, I don't know what the problem could be. I hope you can help me.

Best wishes. Raúl.

[Ruoyu-y]

@rreisim However, the registration of either hub or edge should only use the ip address available in the cnf, in this case, 10.10.70.4x. Since when the two cnfs try to establish ipsec tunnels, they will contact each other using this ip address. May i know what issues do you have while registering with the CNF IP?

[rreisim]

@Ruoyu-y Exactly, when we try to register the edge and the hub with only the cnf ips, central-controller doesn't encounter that ips. However, with the public ips this error disappear.

[Ruoyu-y]

@Ruoyu-y Exactly, when we try to register the edge and the hub with only the cnf ips, central-controller doesn't encounter that ips. However, with the public ips this error disappear.

Can you help paste the error here? Actually we do a cluster connection test before we finished the registration, so the error should be related to that.

[rreisim]

@Ruoyu-y

I think the problem is that it doesn't find the IP of the CNF because the IP is not visible to the local environment. Should the scc also get a net2 IP?

[Ruoyu-y]

@rreisim Can you please go inside your cnf container, and check the iptables there. There should be some DNAT rules to redirect the traffic to api server. So that scc could use this ip address to access the api server. Otherwise, there might be something missing in your cnf helm chart. By the way, have you add something routing rules in your host vm to route the scc request into the cnf container? Something like:

ip r add 10.10.70.0/24 dev <cnf-interface-name>

[rreisim]

@Ruoyu-y Ok thanks for your contribution, by adding the following routing in VM host I manage to add hub and edge with their respective Ips 10.10.70.4x.

ip r add 10.10.70.0/24 via 10.210.77.145 (cnf tunl0 ip)

However, by repeating the above steps with the CNF ips, the subnet is not created on the edge node. I will try to find a solution and will reply to this thread with my progress next week. Again, thanks for the help.

[rreisim]

Hello again.

After several attempts, the central controller is still not working properly. I have the same problem as when I started with this issue. Regardless if the configuration of the public ips are ok, I guess that when the central controller creates the "ipsecsite crd" on the edge node and the "ipsechost crd" on the hub node, if both have communication, the new subnet should be created in the cnf of the egde node.

By the way, I have checked all the configuration and now, I can say that I haven't found any more errors in the configuration. So, I really don't know what is the main problem here, but the subnet is not created when I apply the configuration you can see in first message.

So, can you show me any configuration or deployment you have ready or any PoC you have done before with or without central-controller? Maybe an example can be relevant to finish this issue.

Greetings, Raúl.

[Ruoyu-y]

Okay. Can you try these simplified configurations? These are working on our local environment. This is the one for the hub.

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: IpsecSite
metadata:  
  labels:
    emco/deployment-id: 300632648498745066-huba-app
    sdewanPurpose: base
    targetCluster: Hub.huba
  name: hubaedge1
  namespace: default
spec:
  authentication_method: psk
  connections:
  - conn_type: tunnel
    crypto_proposal:
    - proposal1
    - proposal2
    local_subnet: 0.0.0.0/0
    local_updown: /etc/updown
    mark: "30"
    mode: start
    name: Connedge1_19216905
    remote_sourceip: 192.169.0.1
  crypto_proposal:
  - proposal1
  - proposal2
  force_crypto_proposal: "0"
  local_identifier: CN=hub-huba-cert
  pre_shared_key: "1234"
  remote: '%any'
  remote_identifier: CN=device-edge-1-cert
  type: VTI-based

This is the one on the edge:

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: IpsecHost
metadata:
  labels:
    emco/deployment-id: 6169737821829001716-edge-1-app
    sdewanPurpose: base
    targetCluster: Device.edge-1
  name: edge1huba
  namespace: default
spec:
  authentication_method: psk
  connections:
  - conn_type: tunnel
    crypto_proposal:
    - proposal1
    - proposal2
    local_sourceip: '%config'
    local_updown: /etc/updown_oip
    mode: start
    name: Connhuba_10107039
    remote_subnet: 0.0.0.0/0
  crypto_proposal:
  - proposal1
  - proposal2
  force_crypto_proposal: "1"
  local_identifier: CN=device-edge-1-cert
  pre_shared_key: "1234"
  remote: <the-ip-address-of-the-hub-cnf> //maybe 10.10.70.40
  remote_identifier: CN=hub-huba-cert
  type: policy-based

Once these configurations are applied, you could the ipsec tunnel up like this:

/ $ sudo ipsec statusall
Status of IKE charon daemon (strongSwan 5.9.2, Linux 5.4.0-42-generic, x86_64):
  uptime: 12 minutes, since Nov 28 10:40:42 2022
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 4
  loaded plugins: charon aes des rc2 sha2 sha1 md5 random nonce x509 revocation constraints pubkey pkcs1 pgp dnskey sshkey pem af-alg fips-prf gmp xcbc hmac attr kernel-netlink resolve socket-default connmark stroke updown xauth-generic
Listening IP addresses:
  10.233.112.92
  10.10.70.19
  172.16.70.19
  10.154.142.40
Connections:
edge1huba-Connhuba_10107039:  %any...10.10.70.39  IKEv2, dpddelay=30s
edge1huba-Connhuba_10107039:   local:  [CN=device-edge-1-cert] uses pre-shared key authentication
edge1huba-Connhuba_10107039:   remote: [CN=hub-huba-cert] uses pre-shared key authentication
edge1huba-Connhuba_10107039:   child:  dynamic === 0.0.0.0/0 TUNNEL, dpdaction=restart
Security Associations (1 up, 0 connecting):
edge1huba-Connhuba_10107039[1]: ESTABLISHED 12 minutes ago, 10.10.70.19[CN=device-edge-1-cert]...10.10.70.39[CN=hub-huba-cert]
edge1huba-Connhuba_10107039[1]: IKEv2 SPIs: 646ee384b2479366_i* 927954849783fc2d_r, pre-shared key reauthentication in 2 hours
edge1huba-Connhuba_10107039[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_4096
edge1huba-Connhuba_10107039{1}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: c8488aa4_i ccfb127e_o
edge1huba-Connhuba_10107039{1}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 34 minutes
edge1huba-Connhuba_10107039{1}:   192.169.0.1/32 === 0.0.0.0/0

And you could see the virtual ip address attached as below:

/ $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
5: eth0@if92: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether 1a:65:1b:8a:55:a9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.233.112.92/32 scope global eth0
       valid_lft forever preferred_lft forever
7: net2@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
    link/ether 00:00:00:f3:17:6f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.10.70.19/24 brd 10.10.70.255 scope global net2
       valid_lft forever preferred_lft forever
    inet 192.169.0.1/32 scope global net2
       valid_lft forever preferred_lft forever
    inet6 fe80::200:ff:fef3:176f/64 scope link
       valid_lft forever preferred_lft forever

Please let me know if this works on your side.

[rreisim]

Hello @Ruoyu-y

It works! The configuration appears correctly in net2 in each edge nodes.

bash-5.1$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if333: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether 92:1c:18:13:5b:2c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.210.77.165/32 brd 10.210.77.165 scope global eth0
       valid_lft forever preferred_lft forever
6: net2@if334: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default 
    link/ether 00:00:00:3d:9c:dd brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.10.70.41/24 brd 10.10.70.255 scope global net2
       valid_lft forever preferred_lft forever
    inet 192.168.10.1/32 scope global net2
       valid_lft forever preferred_lft forever
    inet6 fe80::200:ff:fe3d:9cdd/64 scope link 
       valid_lft forever preferred_lft forever
8: net0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default 
    link/ether 00:00:00:93:14:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.16.70.41/24 brd 172.16.70.255 scope global net0
       valid_lft forever preferred_lft forever
    inet6 fe80::200:ff:fe93:14d0/64 scope link 
       valid_lft forever preferred_lft forever
10: net1@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default 
    link/ether f2:a3:4c:97:80:73 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.151.128.114/18 brd 10.151.191.255 scope global net1
       valid_lft forever preferred_lft forever

For my part the issue is solved, thank you very much for your help.

Now, I am having some problems when applying the SNAT firewall because the traffic is not redirected well. Attached is the deployment used.

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: FirewallSNAT
metadata:
  name: firewallsnat
  namespace: default
  labels:
    sdewanPurpose: base
    targetCluster: Device.device2
spec:
  src_ip: 172.16.70.0/24 # subnet net0
  src_dip: 192.168.10.1 # virtual IP edge-a
  dest: pnetwork
  dest_ip: 10.10.70.40 # net2 ip hub
  proto: all
  target: SNAT

[Ruoyu-y]

@rreisim Good news. And for the NAT rules, please use the CNFNAT CRD to construct that, as the FirewallSNAT is going to be deprecated because of its complexity. The sample would be like this:

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: CNFNAT
metadata:
  labels:
    emco/deployment-id: 7176391176289176379-edge-1-app
    sdewanPurpose: base
    targetCluster: Device.edge-1
  name: nat4test
  namespace: default
spec:
  dest: net2 # interface where the traffic goes out. You can also use '#source' instead to get the interface name of the src_dip address
  dest_ip: 10.10.70.40 # the destination ip
  src_ip: 172.16.70.0/24 # the original source ip
  index: "0"   # the index of the rule within the iptables chain, this means the first one
  src_dip: 192.168.10.1 # SNAT result
  target: SNAT

Please let me know any other issues found.

[rreisim]

Well, @Ruoyu-y.

After implementing everything, I still can't get requests from my httpbin pod (edge-a) to my httpbin pod (edge-b).

I have tried with different configurations, concluding that both SNAT and DNAT configured work correctly, but I find an error when I try to perform the final test.

The configuration I have used on edge-a is:

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: CNFNAT
metadata:
  labels:
    sdewanPurpose: base
    targetCluster: Device.device2
  name: cnfsnat
  namespace: default
spec:
  src_ip: 172.16.70.0/24 # the original source ip
  src_dip: 192.168.10.1 # SNAT result
  dest: net2
  dest_ip: 10.10.70.42 # CNF-HUB net2 IP
  index: "0"
  target: SNAT

and edge-b:

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: CNFService
metadata:
  name: cnfservice
  namespace: default
  labels:
    sdewanPurpose: base
    targetCluster: Device.device3
spec:
    fullname: my-http-service.default.svc.cluster.local
    port: "80"
    dport: "80"

I think the error might be related to how the hub intercepts packets and replicates them to edge-b. Sorry for the inconvenience.

[Ruoyu-y]

@rreisim Can you tell me the topology of your environment? And the last configuration of that cnfservice, are you trying to access the service through domain name?

[rreisim]

Hello @Ruoyu-y. I attached you the general topology of my environment. Regarding the cnfservice, in general terms, works correctly for a first approximation.

So, my "APP1: HTTPBIN" request would be:

curl -X GET "http://192.168.10.2/ip" -H "accept: application/json" 

And the expected response from "APP2: HTBBIN" would be:

{
  "origin": "192.168.10.1"
}

[Ruoyu-y]

@rreisim The topology looks fine to me. I think there's no need to add the cnfservice in this case. The SNAT/DNATs are enough. Regarding your issue, please first check that the virtual ips can get connected from both sides using the IPSec tunnel. Try ping 192.168.10.x on both sides. You can also use such command in the CNF container to check the ipsec status: sudo ipsec statusall And i double checked the configuration you used for SNAT, are you trying to change the src ip to 192.168.10.1 when the packet is going to 10.10.70.42? My thinking is that you should have the snat take place when the packet is going to 192.168.10.2 instead of 10.10.70.42. Please correct me if anything wrong.

[rreisim]

Hi @Ruoyu-y thanks for be persisent with this issue.

Unfortunately I haven't connection between edge nodes. I don't receive response after pinging between them. The sudo ipsec statusall is the next:

Hub:

bash-5.1$ sudo ipsec statusall
Status of IKE charon daemon (strongSwan 5.9.2, Linux 5.15.0-46-generic, x86_64):
  uptime: 27 minutes, since Dec 07 10:54:23 2022
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 51
  loaded plugins: charon aes des rc2 sha2 sha1 md5 random nonce x509 revocation constraints pubkey pkcs1 pgp dnskey sshkey pem af-alg fips-prf gmp xcbc hmac attr kernel-netlink resolve socket-default connmark stroke updown xauth-generic
Virtual IP pools (size/online/offline):
  192.168.10.1: 1/1/0
  192.168.10.2: 1/1/0
Listening IP addresses:
  10.210.77.154
  10.10.70.40
  172.16.70.40
  10.151.128.88
Connections:
hub1device2-Conndevice2_192168101:  %any...%any  IKEv2, dpddelay=30s
hub1device2-Conndevice2_192168101:   local:  [CN=hub-hub1-cert] uses pre-shared key authentication
hub1device2-Conndevice2_192168101:   remote: [CN=device-device2-cert] uses pre-shared key authentication
hub1device2-Conndevice2_192168101:   child:  0.0.0.0/0 === dynamic TUNNEL, dpdaction=restart
hub1device3-Conndevice3_192168102:  %any...%any  IKEv2, dpddelay=30s
hub1device3-Conndevice3_192168102:   local:  [CN=hub-hub1-cert] uses pre-shared key authentication
hub1device3-Conndevice3_192168102:   remote: [CN=device-device3-cert] uses pre-shared key authentication
hub1device3-Conndevice3_192168102:   child:  0.0.0.0/0 === dynamic TUNNEL, dpdaction=restart
Security Associations (2 up, 0 connecting):
hub1device3-Conndevice3_192168102[26]: ESTABLISHED 10 minutes ago, 10.10.70.40[CN=hub-hub1-cert]...10.10.70.42[CN=device-device3-cert]
hub1device3-Conndevice3_192168102[26]: IKEv2 SPIs: 5588b29a6cd6dc67_i 50ccfbf6a20edbf3_r*, pre-shared key reauthentication in 2 hours
hub1device3-Conndevice3_192168102[26]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_4096
hub1device3-Conndevice3_192168102[26]: Tasks active: IKE_DPD 
hub1device3-Conndevice3_192168102{33}:  INSTALLED, TUNNEL, reqid 2, ESP SPIs: cf383ef2_i c269da44_o
hub1device3-Conndevice3_192168102{33}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 33 minutes
hub1device3-Conndevice3_192168102{33}:   0.0.0.0/0 === 192.168.10.2/32
hub1device3-Conndevice3_192168102{34}:  INSTALLED, TUNNEL, reqid 2, ESP SPIs: c6af21e9_i ca3eaf32_o
hub1device3-Conndevice3_192168102{34}:  AES_CBC_256/HMAC_SHA2_256_128/MODP_4096, 0 bytes_i, 0 bytes_o, rekeying in 33 minutes
hub1device3-Conndevice3_192168102{34}:   0.0.0.0/0 === 192.168.10.2/32
hub1device2-Conndevice2_192168101[25]: ESTABLISHED 11 minutes ago, 10.10.70.40[CN=hub-hub1-cert]...10.10.70.41[CN=device-device2-cert]
hub1device2-Conndevice2_192168101[25]: IKEv2 SPIs: 37bd81960b1707ed_i 329b06929a2fbb3f_r*, pre-shared key reauthentication in 2 hours
hub1device2-Conndevice2_192168101[25]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_4096
hub1device2-Conndevice2_192168101{31}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: cef38a81_i c0edb3ba_o
hub1device2-Conndevice2_192168101{31}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 32 minutes
hub1device2-Conndevice2_192168101{31}:   0.0.0.0/0 === 192.168.10.1/32
hub1device2-Conndevice2_192168101{32}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: c6e2c209_i c048cbdf_o
hub1device2-Conndevice2_192168101{32}:  AES_CBC_256/HMAC_SHA2_256_128/MODP_4096, 0 bytes_i, 0 bytes_o, rekeying in 31 minutes
hub1device2-Conndevice2_192168101{32}:   0.0.0.0/0 === 192.168.10.1/32

edge-a:

bash-5.1$ sudo ipsec statusall
Status of IKE charon daemon (strongSwan 5.9.2, Linux 5.15.0-56-generic, x86_64):
  uptime: 11 minutes, since Dec 07 11:10:39 2022
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 4
  loaded plugins: charon aes des rc2 sha2 sha1 md5 random nonce x509 revocation constraints pubkey pkcs1 pgp dnskey sshkey pem af-alg fips-prf gmp xcbc hmac attr kernel-netlink resolve socket-default connmark stroke updown xauth-generic
Listening IP addresses:
  10.210.77.132
  10.10.70.41
  172.16.70.41
  10.151.128.128
Connections:
device2hub1-Connhub1_10107040:  %any...10.10.70.40  IKEv2, dpddelay=30s
device2hub1-Connhub1_10107040:   local:  [CN=device-device2-cert] uses pre-shared key authentication
device2hub1-Connhub1_10107040:   remote: [CN=hub-hub1-cert] uses pre-shared key authentication
device2hub1-Connhub1_10107040:   child:  dynamic === 0.0.0.0/0 TUNNEL, dpdaction=restart
Security Associations (1 up, 0 connecting):
device2hub1-Connhub1_10107040[1]: ESTABLISHED 11 minutes ago, 10.10.70.41[CN=device-device2-cert]...10.10.70.40[CN=hub-hub1-cert]
device2hub1-Connhub1_10107040[1]: IKEv2 SPIs: 37bd81960b1707ed_i* 329b06929a2fbb3f_r, pre-shared key reauthentication in 2 hours
device2hub1-Connhub1_10107040[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_4096
device2hub1-Connhub1_10107040{1}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: c0edb3ba_i cef38a81_o
device2hub1-Connhub1_10107040{1}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 35 minutes
device2hub1-Connhub1_10107040{1}:   192.168.10.1/32 === 0.0.0.0/0
device2hub1-Connhub1_10107040{2}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: c048cbdf_i c6e2c209_o
device2hub1-Connhub1_10107040{2}:  AES_CBC_256/HMAC_SHA2_256_128/MODP_4096, 0 bytes_i, 0 bytes_o, rekeying in 33 minutes
device2hub1-Connhub1_10107040{2}:   192.168.10.1/32 === 0.0.0.0/0

edge-b:

bash-5.1$ sudo ipsec statusall
Status of IKE charon daemon (strongSwan 5.9.2, Linux 5.15.0-41-generic, x86_64):
  uptime: 11 minutes, since Dec 07 11:11:15 2022
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 4
  loaded plugins: charon aes des rc2 sha2 sha1 md5 random nonce x509 revocation constraints pubkey pkcs1 pgp dnskey sshkey pem af-alg fips-prf gmp xcbc hmac attr kernel-netlink resolve socket-default connmark stroke updown xauth-generic
Listening IP addresses:
  10.210.151.132
  10.10.70.42
  172.16.70.42
  10.151.128.83
Connections:
device3hub1-Connhub1_10107040:  %any...10.10.70.40  IKEv2, dpddelay=30s
device3hub1-Connhub1_10107040:   local:  [CN=device-device3-cert] uses pre-shared key authentication
device3hub1-Connhub1_10107040:   remote: [CN=hub-hub1-cert] uses pre-shared key authentication
device3hub1-Connhub1_10107040:   child:  dynamic === 0.0.0.0/0 TUNNEL, dpdaction=restart
Security Associations (1 up, 0 connecting):
device3hub1-Connhub1_10107040[1]: ESTABLISHED 11 minutes ago, 10.10.70.42[CN=device-device3-cert]...10.10.70.40[CN=hub-hub1-cert]
device3hub1-Connhub1_10107040[1]: IKEv2 SPIs: 5588b29a6cd6dc67_i* 50ccfbf6a20edbf3_r, pre-shared key reauthentication in 2 hours
device3hub1-Connhub1_10107040[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_4096
device3hub1-Connhub1_10107040{1}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: c269da44_i cf383ef2_o
devConnhub1_10107040{1}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 33 minutes
device3hub1-Connhub1_10107040{1}:   192.168.10.2/32 === 0.0.0.0/0
device3hub1-Connhub1_10107040{2}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: ca3eaf32_i c6af21e9_o
device3hub1-Connhub1_10107040{2}:  AES_CBC_256/HMAC_SHA2_256_128/MODP_4096, 0 bytes_i, 0 bytes_o, rekeying in 33 minutes
device3hub1-Connhub1_10107040{2}:   192.168.10.2/32 === 0.0.0.0/0

Is it necessary to create another CRD (ipsecsite or ipsechost) to connect the two nodes or should the hub do it immediately?

By the way, it is also not possible to ping between the hub and any edge node.

[Ruoyu-y]

@rreisim. There's no need to add extra ipsec CR for the connection. Can you try to use sudo nsenter -t <pid-of-cnf> -n ip xfrm m to monitor your pings and see if there's traffic showing there. Also, please use tcpdump to check the traffic and see if it goes into the right interface.

[rreisim]

I close this thread because it was possible to establish a connection through the IP tunne