Dai-Yi commented 1 year ago

Expected Behavior

EIP can ping, curl EIP which bound to Pod running nginx could get response

Actual Behavior

EIP can‘t ping, curl EIP which bound to Pod running nginx get hint: “No route to host” or “Timeout”

Steps to Reproduce the Problem

1.apply configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: ovn-external-gw-config
  namespace: kube-system
data:
  enable-external-gw: "true"
  type: "centrailized"
  external-gw-nodes: "eb62167"
  external-gw-nic: "ens35"
  external-gw-addr: "10.1.61.112/24"
  nic-ip: "10.1.61.100/24"
  nic-mac: "16:52:f3:13:6a:25"

2.Confirm the Configuration Take Effect

# kubectl ko nbctl show
switch f087444c-34c7-40ca-a6e4-a290b4798dbf (ovn-external)
    port ln-ovn-external
        type: localnet
        addresses: ["unknown"]
    port ovn-external-ovn-cluster
        type: router
        router-port: ovn-cluster-ovn-external
router 4a6d2185-ed98-4a8c-b15f-773bd346780a (ovn-cluster)
    port ovn-cluster-ovn-external
        mac: "16:52:f3:13:6a:25"
        networks: ["10.1.61.100/24"]
        gateway chassis: [d17fc2cb-6d47-4ad3-b10d-c14dbe450cea]
# kubectl ko vsctl eb62167 show
1310e607-d9d2-49c0-963f-b84b433875d3
    Bridge br-external
        Port br-external
            Interface br-external
                type: internal
        Port ens35
            Interface ens35
        Port patch-ln-ovn-external-to-br-int
            Interface patch-ln-ovn-external-to-br-int
                type: patch
                options: {peer=patch-br-int-to-ln-ovn-external}

3.Config EIP on Pod

apiVersion: v1
kind: Pod
metadata:
  name: pod-gw-eip
  annotations:
    ovn.kubernetes.io/eip: 10.1.61.200
spec:
  containers:
  - name: eip-pod
    image: nginx
    ports:
    -  name: nginx-port
       containerPort: 80

Additional Info

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.20", GitCommit:"1f3e19b7beb1cc0110255668c4238ed63dadb7ad", GitTreeState:"clean", BuildDate:"2021-06-16T12:51:17Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

kube-ovn version:
```
v1.10.7
```

operation-system/kernel version:

CentOS Linux 7 (Core)
3.10.0-957.el7.x86_64

kube-ovn configuration parameter:

--default-cidr=10.16.0.0/16
--default-gateway=10.16.0.1
--default-gateway-check=true
--default-logical-gateway=false
--default-exclude-ips=
--node-switch-cidr=100.64.0.0/16
--service-cluster-ip-range=10.96.0.0/12
--network-type=geneve
--default-interface-name=
--default-vlan-id=100
--pod-nic-type=veth-pair
--enable-lb=true
--enable-np=true
--enable-eip-snat=true
--enable-external-vpc=true
--logtostderr=false
--alsologtostderr=true
--log_file=/var/log/kube-ovn/kube-ovn-controller.log
--log_file_max_size=0
--keep-vm-ip=true

OVN-NB status after apply pod

switch f087444c-34c7-40ca-a6e4-a290b4798dbf (ovn-external)
port ln-ovn-external
    type: localnet
    addresses: ["unknown"]
port ovn-external-ovn-cluster
    type: router
    router-port: ovn-cluster-ovn-external
switch 7119020a-4b1b-4300-a4d5-69a514bd5b96 (ovn-default)
port pod-gw-eip.default
    addresses: ["00:00:00:6A:C5:CF 10.16.0.23"]
port metrics-server-555ff645f9-757lb.kube-system
    addresses: ["00:00:00:34:A2:9C 10.16.0.9"]
port coredns-7ff77c879f-r45lv.kube-system
    addresses: ["00:00:00:05:BE:00 10.16.0.7"]
port kube-ovn-pinger-vbc4z.kube-system
    addresses: ["00:00:00:BE:55:88 10.16.0.8"]
port ovn-default-ovn-cluster
    type: router
    router-port: ovn-cluster-ovn-default
port nginx.default
    addresses: ["00:00:00:FA:52:ED 10.16.0.10"]
port coredns-7ff77c879f-zm2hz.kube-system
    addresses: ["00:00:00:CF:8F:94 10.16.0.6"]
switch bb6a2707-a4a0-4b89-9f27-a56934cbcec9 (join)
port join-ovn-cluster
    type: router
    router-port: ovn-cluster-join
port node-eb62167
    addresses: ["00:00:00:EF:24:6E 100.64.0.2"]
router 4a6d2185-ed98-4a8c-b15f-773bd346780a (ovn-cluster)
port ovn-cluster-ovn-external
    mac: "16:52:f3:13:6a:25"
    networks: ["10.1.61.100/24"]
    gateway chassis: [d17fc2cb-6d47-4ad3-b10d-c14dbe450cea]
port ovn-cluster-ovn-default
    mac: "00:00:00:03:C5:AC"
    networks: ["10.16.0.1/16"]
port ovn-cluster-join
    mac: "00:00:00:19:CF:DD"
    networks: ["100.64.0.1/16"]
nat ff8a405b-64fd-459d-802d-958387e0686e
    external ip: "10.1.61.200"
    logical ip: "10.16.0.23"
    type: "dnat_and_snat"

OVS status after apply pod

1310e607-d9d2-49c0-963f-b84b433875d3
Bridge br-external
    Port br-external
        Interface br-external
            type: internal
    Port ens35
        Interface ens35
    Port patch-ln-ovn-external-to-br-int
        Interface patch-ln-ovn-external-to-br-int
            type: patch
            options: {peer=patch-br-int-to-ln-ovn-external}
Bridge br-int
    fail_mode: secure
    datapath_type: system
    Port mirror0
        Interface mirror0
            type: internal
    Port "4e5a8c3cd1a6_h"
        Interface "4e5a8c3cd1a6_h"
    Port patch-br-int-to-ln-ovn-external
        Interface patch-br-int-to-ln-ovn-external
            type: patch
            options: {peer=patch-ln-ovn-external-to-br-int}
    Port ovn0
        Interface ovn0
            type: internal
    Port fe399548918c_h
        Interface fe399548918c_h
    Port "8529624a104b_h"
        Interface "8529624a104b_h"
    Port a0c63ab7880c_h
        Interface a0c63ab7880c_h
    Port "595121e1c9c9_h"
        Interface "595121e1c9c9_h"
    Port f78e5dedcfed_h
        Interface f78e5dedcfed_h
    Port br-int
        Interface br-int
            type: internal
ovs_version: "2.17.4"

ifconfig (partly)

ens32: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 10.1.62.167  netmask 255.255.255.0  broadcast 10.1.62.255
    inet6 2001:470:8287:602:472b:3a25:d057:e88c  prefixlen 64  scopeid 0x0<global>
    inet6 fe80::79b2:6929:4578:b9c4  prefixlen 64  scopeid 0x20<link>
    ether 00:50:56:a6:25:15  txqueuelen 1000  (Ethernet)
    RX packets 5774097787  bytes 930541915330 (866.6 GiB)
    RX errors 0  dropped 27718  overruns 0  frame 0
    TX packets 5064160368  bytes 917438332541 (854.4 GiB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
ens35: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 10.1.61.112  netmask 255.255.255.0  broadcast 10.1.61.255
    inet6 2001:470:8287:602:9c5:c728:93ef:aaa6  prefixlen 64  scopeid 0x0<global>
    inet6 2001:470:8287:601:1a25:ad:459e:b3a9  prefixlen 64  scopeid 0x0<global>
    ether 00:50:56:a6:c0:28  txqueuelen 1000  (Ethernet)
    RX packets 5211118  bytes 387046423 (369.1 MiB)
    RX errors 0  dropped 23352  overruns 0  frame 0
    TX packets 161893  bytes 19881182 (18.9 MiB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
ens36: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 10.1.35.38  netmask 255.255.255.0  broadcast 10.1.35.255
    inet6 2001:470:8287:602:bd33:73ea:68a0:c525  prefixlen 64  scopeid 0x0<global>
    inet6 fe80::f4fa:ffb8:cd67:e428  prefixlen 64  scopeid 0x20<link>
    inet6 2001:470:8287:305:e95a:733f:ba77:d364  prefixlen 64  scopeid 0x0<global>
    ether 00:50:56:a6:4f:9a  txqueuelen 1000  (Ethernet)
    RX packets 28928693  bytes 1780585179 (1.6 GiB)
    RX errors 0  dropped 26059  overruns 0  frame 0
    TX packets 160026  bytes 20026812 (19.0 MiB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

希望老哥帮忙看看怎么回事

bobz965 commented 1 year ago

同样地，目前master分支，在集群内部的node上ping eip 是ok的，但是telnet 和 curl 端口应该是有问题的，集群外部ping 和 curl应该都是ok的吧？

Dai-Yi commented 1 year ago

同样地，目前master分支，在集群内部的node上ping eip 是ok的，但是telnet 和 curl 端口应该是有问题的，集群外部ping 和 curl应该都是ok的吧？

集群内集群外ping EIP都ping不通，我感觉配置EIP的configmap和Pod的annotations是不是没配对

bobz965 commented 1 year ago

同样地，目前master分支，在集群内部的node上ping eip 是ok的，但是telnet 和 curl 端口应该是有问题的，集群外部ping 和 curl应该都是ok的吧？

集群内集群外ping EIP都ping不通，我感觉配置EIP的configmap和Pod的annotations是不是没配对

external-gw-addr: "10.1.61.112/24" 这个不是真实的物理网关吧？

Dai-Yi commented 1 year ago

同样地，目前master分支，在集群内部的node上ping eip 是ok的，但是telnet 和 curl 端口应该是有问题的，集群外部ping 和 curl应该都是ok的吧？

集群内集群外ping EIP都ping不通，我感觉配置EIP的configmap和Pod的annotations是不是没配对

external-gw-addr: "10.1.61.112/24" 这个不是真实的物理网关吧？

10.1.61.112这是节点上承担网关作用的那个网卡ens35的ip地址，请问我的物理网关地址是以下显示的第一个（10.1.62.1）吗？

route -n

Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.1.62.1 0.0.0.0 UG 0 0 0 ens32 10.1.35.0 0.0.0.0 255.255.255.0 U 101 0 0 ens36 10.1.61.0 0.0.0.0 255.255.255.0 U 100 0 0 ens35 10.1.62.0 0.0.0.0 255.255.255.0 U 0 0 0 ens32 10.16.0.0 100.64.0.1 255.255.0.0 UG 0 0 0 ovn0 100.64.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ovn0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens32 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 我把configmap中的external-gw-addr项改为10.1.62.1后仍然ping不通EIP

bobz965 commented 1 year ago

同样地，目前master分支，在集群内部的node上ping eip 是ok的，但是telnet 和 curl 端口应该是有问题的，集群外部ping 和 curl应该都是ok的吧？

集群内集群外ping EIP都ping不通，我感觉配置EIP的configmap和Pod的annotations是不是没配对

external-gw-addr: "10.1.61.112/24" 这个不是真实的物理网关吧？

10.1.61.112这是节点上承担网关作用的那个网卡ens35的ip地址，请问我的物理网关地址是以下显示的第一个（10.1.62.1）吗？

route -n

Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.1.62.1 0.0.0.0 UG 0 0 0 ens32 10.1.35.0 0.0.0.0 255.255.255.0 U 101 0 0 ens36 10.1.61.0 0.0.0.0 255.255.255.0 U 100 0 0 ens35 10.1.62.0 0.0.0.0 255.255.255.0 U 0 0 0 ens32 10.16.0.0 100.64.0.1 255.255.0.0 UG 0 0 0 ovn0 100.64.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ovn0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens32 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 我把configmap中的external-gw-addr项改为10.1.62.1后仍然ping不通EIP

我只用过物理交换机的真实网关的场景，感觉真实物理网关是没问题的。

github-actions[bot] commented 11 months ago

Issues go stale after 60d of inactivity. Please comment or re-open the issue if you are still interested in getting this issue fixed.

kubeovn / kube-ovn

pod EIP can’t work under defualt VPC #2051

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Additional Info

route -n

route -n