search

kubeovn / kube-ovn

A Bridge between SDN and Cloud Native (Project under CNCF)
https://kubeovn.github.io/docs/stable/en/
Apache License 2.0
1.93k stars 438 forks source link

EIP NAT not working for few IPs in subnet #3531

Closed nics90 closed 6 months ago

nics90 commented 9 months ago

Bug Report

We are using kube-ovn in our production environment where we have created subnets and used EIP feature to expose Kubevirt VMs to external network. We are seeing the issue where EIP worked for few days but suddenly stopped working for some random VMs (Means EIP IP which is a public IP is not reachable through internet). We did below steps:

a) We checked the NAT entries using kubectl ko nbctl show - It looks fine since the entry is present. b) Private IP assigned by subnet is pinging from the host cluster. c) EIP is not reachable through internet. d) We tried changing the interface on the VM as well as migrating them to some other host in the cluster but nothing works.

Expected Behavior

EIP should be accessible from outside cluster.

Actual Behavior

EIP is not reachable

Additional Info

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-04-13T19:57:43Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-04-13T19:52:02Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

# kubectl get deployment -n kube-system kube-ovn-controller -o yaml | grep image:
        image: kubeovn/kube-ovn:v1.11.3

NAT Output:

~# kubectl ko nbctl show | grep -b1 -A2  147.x.x.138
34688-    nat 424cb867-aca2-449a-9496-2ad3a4d2cff6
34733:        external ip: "147.x.x.138"
34772-        logical ip: "10.40.45.3"
34805-        type: "dnat_and_snat"

Vsctl Outout:


kubectl ko vsctl server-cl-sr12 show
1b92e000-6196-4279-81ce-71257a5dc190
    Bridge br-la-pv-950
        Port br-la-pv-950
            Interface br-la-pv-950
                type: internal
        Port veth53e8a347
            Interface veth53e8a347
        Port bond0.950
            Interface bond0.950
    Bridge br-mh-pv-nw
        Port bond0.1818
            Interface bond0.1818
        Port br-mh-pv-nw
            Interface br-mh-pv-nw
                type: internal
    Bridge br-la-pv-1198
        Port bond0.1198
            Interface bond0.1198
        Port br-la-pv-1198
            Interface br-la-pv-1198
                type: internal
    Bridge br-la-pv-922
        Port bond0.922
            Interface bond0.922
        Port br-la-pv-922
            Interface br-la-pv-922
                type: internal
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port "68005dab7ff5_h"
            Interface "68005dab7ff5_h"
        Port "6ee117ff3041_h"
            Interface "6ee117ff3041_h"
        Port ovn0
            Interface ovn0
                type: internal
        Port "69f72014d6b9_h"
            Interface "69f72014d6b9_h"
        Port "0da4a05541dc_h"
            Interface "0da4a05541dc_h"
        Port "159fb91d5250_h"
            Interface "159fb91d5250_h"
        Port "0c7633d3f249_h"
            Interface "0c7633d3f249_h"
        Port "3c651423e0cc_h"
            Interface "3c651423e0cc_h"
        Port aaeadf69303c_h
            Interface aaeadf69303c_h
        Port "92fb4aca248c_h"
            Interface "92fb4aca248c_h"
        Port "7b9f55b6c46d_h"
            Interface "7b9f55b6c46d_h"
        Port br-int
            Interface br-int
                type: internal
        Port "3995cc775891_h"
            Interface "3995cc775891_h"
        Port ovn-2148ee-0
            Interface ovn-2148ee-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.86.0.14"}
        Port "55426faa238d_h"
            Interface "55426faa238d_h"
        Port c21dfef5b40c_h
            Interface c21dfef5b40c_h
        Port dd6c477712f8_h
            Interface dd6c477712f8_h
        Port ovn-8d355d-0
            Interface ovn-8d355d-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.86.0.18"}
        Port "3876f5446261_h"
            Interface "3876f5446261_h"
        Port "30be781fa1e7_h"
            Interface "30be781fa1e7_h"
        Port ovn-63e98b-0
            Interface ovn-63e98b-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.86.0.17"}
        Port "2218f64224ec_h"
            Interface "2218f64224ec_h"
        Port "0669cc824a0e_h"
            Interface "0669cc824a0e_h"
        Port "4a26e839f601_h"
            Interface "4a26e839f601_h"
        Port "91d07753c141_h"
            Interface "91d07753c141_h"
        Port "1d341ada603c_h"
            Interface "1d341ada603c_h"
        Port "1b31068adc86_h"
            Interface "1b31068adc86_h"
        Port "575762459510_h"
            Interface "575762459510_h"
        Port mirror0
            Interface mirror0
                type: internal
        Port df68a007035e_h
            Interface df68a007035e_h
        Port "045a9a064ea4_h"
            Interface "045a9a064ea4_h"
        Port "3051da717aee_h"
            Interface "3051da717aee_h"
        Port ovn-4b144c-0
            Interface ovn-4b144c-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.86.0.19"}
        Port d8975598_net1_h
            Interface d8975598_net1_h
        Port c5ece262c7ad_h
            Interface c5ece262c7ad_h
        Port d774b30b17a1_h
            Interface d774b30b17a1_h
        Port "0f2f33fb3f8a_h"
            Interface "0f2f33fb3f8a_h"
        Port d8975598ba0d_h
            Interface d8975598ba0d_h
        Port "18defeb8b4cb_h"
            Interface "18defeb8b4cb_h"
        Port f1082939f8bb_h
            Interface f1082939f8bb_h
        Port d8122b988594_h
            Interface d8122b988594_h
        Port c2cf6600b6bc_h
            Interface c2cf6600b6bc_h
        Port b8308768f354_h
            Interface b8308768f354_h
        Port "5aacd3be4923_h"
            Interface "5aacd3be4923_h"
        Port "88e14160ab12_h"
            Interface "88e14160ab12_h"
        Port "09bf5104395d_h"
            Interface "09bf5104395d_h"
        Port f015c4451c0b_h
            Interface f015c4451c0b_h
        Port "757682430de4_h"
            Interface "757682430de4_h"
        Port "6de17117e8cb_h"
            Interface "6de17117e8cb_h"
        Port "3ee3f1dbdb82_h"
            Interface "3ee3f1dbdb82_h"
    ovs_version: "2.17.7"

Config Map:


apiVersion: v1
data:
  enable-external-gw: "true"
  external-gw-addr: 147.x.x.1
  external-gw-nic: bond0.200
  external-gw-nodes: server-cl-sr11
  nic-ip: 147.x.x.x/24
  nic-mac: 56:94:45:59:1a:39
kind: ConfigMap
metadata:
  name: ovn-external-gw-config
  namespace: kube-system
bobz965 commented 9 months ago

Hi， it looks like you are using ovn eip dnat snat fip. you can use the release-1.12-mc branch. we fix many bugs about the ovn eip dnat snat fip in master, but we can not merge all of them into release-1.12. so we provide an independent release-1.12-mc.

you can try release-1.12-mc or master. I think the ovn eip dnat snat fip should work well in custom vpc.

bobz965 commented 9 months ago

Recommendations：

nics90 commented 9 months ago

We are very much interested to use kube ovn version 1.12.x but the documentation says that it requires Kubernetes Version >= 1.23, but we are using Kubernetes Version 1.22.9 in our production environment. Is it really a mandate to use Kubernetes Version >= 1.23 or I can try to install Kube-ovn 1.12.x on Kubernetes Version 1.22.9 ?

bobz965 commented 9 months ago

In my opinion, you can give it a try to install Kube-ovn 1.12.x on Kubernetes Version 1.22.9

github-actions[bot] commented 7 months ago

Issues go stale after 60d of inactivity. Please comment or re-open the issue if you are still interested in getting this issue fixed.