search

kubeovn / kube-ovn

A Bridge between SDN and Cloud Native (Project under CNCF)
https://kubeovn.github.io/docs/stable/en/
Apache License 2.0
1.96k stars 447 forks source link

[BUG] Ping not working between VMs in different subnets via VPC #4680

Open rrajendran17 opened 1 week ago

rrajendran17 commented 1 week ago

Kube-OVN Version

v1.12.28

Kubernetes Version

v1.29.9

Operation-system/Kernel Version

suse/sle-micro/base-5.5:2.0.4-5.8.105

Description

Install kubeovn using install.sh provided

Check the pods running

kubectl get pods -n kube-system -o wide | grep ovn
kube-ovn-cni-rz6rg                                      1/1     Running     0             14d   172.19.98.225   renuka-dev-v131   <none>           <none>
kube-ovn-controller-85b45f7694-tbk9g                    1/1     Running     0             14d   172.19.98.225   renuka-dev-v131   <none>           <none>
kube-ovn-monitor-7c45cd8dcd-x2mqh                       1/1     Running     0             14d   172.19.98.225   renuka-dev-v131   <none>           <none>
kube-ovn-pinger-wt48g                                   1/1     Running     0             14d   10.52.0.107     renuka-dev-v131   <none>           <none>
ovn-central-844fdf4f4f-n6npg                            1/1     Running     0             14d   172.19.98.225   renuka-dev-v131   <none>           <none>
ovs-ovn-c4k27                                           1/1     Running     0             14d   172.19.98.225   renuka-dev-v131   <none>           <none>

2.create vpc

kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: commonvpc

3.create nad attachnet

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: attachnet
  namespace: default
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "kube-ovn",
      "bridge":"cluster-1-br",
      "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
      "provider": "attachnet.default.ovn"
    }'

4.create another nad attachnet2

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: attachnet2
  namespace: default
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "kube-ovn",
      "bridge":"cluster-1-br",
      "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
      "provider": "attachnet2.default.ovn"
    }'

5.create subnet using attachnet nad and attaching to commonvpc vpc

apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: attachnet
spec:
  protocol: IPv4
  provider: attachnet.default.ovn
  vpc: commonvpc
  cidrBlock: 172.20.0.0/16
  gateway: 172.20.0.1
  excludeIps:
  - 172.20.0.0..172.20.0.10

6.create subnet using attachnet2 nad and attaching to commonvpc vpc

apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: attachnet2
spec:
  protocol: IPv4
  provider: attachnet2.default.ovn
  vpc: commonvpc
  cidrBlock: 172.30.0.0/16
  gateway: 172.30.0.1
  excludeIps:
  - 172.30.0.0..172.30.0.10

7.kubectl get subnets

NAME          PROVIDER                 VPC           PROTOCOL   CIDR            PRIVATE   NAT     DEFAULT   GATEWAYTYPE   V4USED   V4AVAILABLE   V6USED   V6AVAILABLE   EXCLUDEIPS                    U2OINTERCONNECTIONIP
attachnet     attachnet.default.ovn    commonvpc     IPv4       172.20.0.0/16   false     false   false     distributed   0        65524         0        0             ["172.20.0.0..172.20.0.10"]   
attachnet2    attachnet2.default.ovn   commonvpc     IPv4       172.30.0.0/16   false     false   false     distributed   0        65524         0        0             ["172.30.0.0..172.30.0.10"]   
join          ovn                      ovn-cluster   IPv4       100.64.0.0/16   false     false   false     distributed   1        65532         0        0             ["100.64.0.1"]                
ovn-default   ovn                      ovn-cluster   IPv4       10.16.0.0/16    false     true    true      distributed   51       65482         0        0             ["10.16.0.1"]

8.kubectl ko nbctl show attachnet

switch 079042cc-2cf9-49ea-aec4-4e2bc460def2 (attachnet)
    port vm1-attachnet.default.attachnet.default.ovn
        addresses: ["ee:19:f5:ac:2d:c4 172.20.0.18"]
    port attachnet-commonvpc
        type: router
        addresses: ["ea:fe:78:15:a3:b8 172.20.0.1/16"]
        router-port: commonvpc-attachnet
    port vm2-attachnet.default.attachnet.default.ovn
        addresses: ["d2:41:04:f2:ae:d1 172.20.0.19"]

9.kubectl ko nbctl show attachnet2

switch 6af21827-4689-4c9f-a34b-1e6a3b96013b (attachnet2)
    port vm1-attachnet2.default.attachnet2.default.ovn
        addresses: ["86:b8:ff:71:2e:6a 172.30.0.13"]
    port attachnet2-commonvpc
        type: router
        addresses: ["d6:4d:83:f5:49:7c 172.30.0.1/16"]
        router-port: commonvpc-attachnet2

10.Login to ovs container to check bridge interfaces

$ kubectl exec -it ovs-ovn-c4k27 -n kube-system /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@renuka-dev-v131:/kube-ovn# ovs-vsctl show       
b62baca6-181d-4ef2-9f22-3b97887b2c1d
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port ovn0
            Interface ovn0
                type: internal
        Port "2ff4d_37a8eec_h"
            Interface "2ff4d_37a8eec_h"
        Port "46372_37a8eec_h"
            Interface "46372_37a8eec_h"
        Port "50f18_37a8eec_h"
            Interface "50f18_37a8eec_h"
        Port br-int
            Interface br-int
                type: internal
        Port mirror0
            Interface mirror0
                type: internal

1.create a vm and attach to attachnet nad.(172.20.0.18/16) 2.create a vm and attach to attachnet nad.(172.20.0.19/16) 3.create a vm and attach to attachnet2 nad(172.30.0.13/16) 4.Ping between 172.20.0.18 to 172.20.0.19 works (same subnet - L2 works) 5.Ping between 172.20.0.18 to 172.30.0.13 fails (different subnet - L3 fails)

Though nbctl shows correct entries for both vm interface and router interface, the ping fails. Is anymore additional config required to achieve this.

Note:This is a single host and all vms are created on the same host.

Steps To Reproduce

1.create a vm and attach to attachnet nad.(172.20.0.18/16) 2.create a vm and attach to attachnet nad.(172.20.0.19/16) 3.create a vm and attach to attachnet2 nad(172.30.0.13/16) 4.Ping between 172.20.0.18 to 172.20.0.19 works (same subnet - L2 works) 5.Ping between 172.20.0.18 to 172.30.0.13 fails (different subnet - L3 fails)

Current Behavior

Ping fails between interfaces of VMs in different subnet via vpc

Expected Behavior

Ping should work between interfaces of VMs in different subnet via vpc.

dosubot[bot] commented 1 week ago

To resolve the issue of ping not working between VMs in different subnets via a VPC in Kube-OVN, you should ensure that the logical router ports are correctly configured to enable inter-subnet communication. Here are some steps and checks you can perform:

  1. Router Port Configuration: Verify that the logical router ports are correctly set up to connect the subnets. The router-port configuration is crucial for enabling communication between different subnets by creating logical router ports that connect logical routers. Ensure that the router ports are correctly defined and linked as peers between the subnets [1].

  2. Routing Table Entries: Check the routing table entries for the VPC to ensure that static and policy routes are correctly configured. The handleAddOrUpdateVpc function manages these routes, and any discrepancies could lead to connectivity issues [2].

  3. Firewall and Security Group Settings: Although not explicitly mentioned in the provided information, ensure that there are no firewall or security group settings that could be blocking inter-subnet traffic.

  4. Logs and Debugging Tools: Use the available logs and debugging tools to diagnose the issue. Commands like kubectl ko nbctl show, kubectl ko sbctl show, and kubectl ko trace can help identify where the connectivity is failing [3].

  5. Single Host Consideration: Since all VMs are on a single host, ensure that the host's network configuration allows for proper routing between the subnets. Sometimes, host-level network settings can interfere with the expected behavior of the overlay network.

By following these steps, you should be able to identify and resolve the issue preventing inter-subnet communication.

To continue talking to Dosu, mention @dosu.

zhangzujian commented 1 week ago

Please provide more details in a readable format, including the VM pod yamls.

rrajendran17 commented 5 days ago

@zhangzujian I have changed the format.And attached a yaml file for one of the vms in attachnet.Please let me know if its fine and need more info, vm1-attachnet.tar.gz

zhangzujian commented 2 days ago

Please provide the Pod yamls instead of the VM yaml.

zhangzujian commented 2 days ago

Also, you can execute ip route get <PING_TARGET_IP> in the VMs to see which route is selected.

rrajendran17 commented 22 hours ago

@zhangzujian currently my setup in not installed with kubeovn and reprodcued state, I will get back to you with the requested outputs in a couple of days, Thanks