search

vmware-archive / kubernetes-archived

This repository is archived. Please file in-tree vSphere Cloud Provider issues at https://github.com/kubernetes/kubernetes/issues . CSI Driver for vSphere is available at https://github.com/kubernetes/cloud-provider-vsphere
Apache License 2.0
46 stars 31 forks source link

Flapping HostIP when node has multiple IP addresses #519

Open Reamer opened 5 years ago

Reamer commented 5 years ago

What happened: Using Openshift with vsphere cloud provider, works as expected till you add multiple egressIPs. After adding multiple EGRESS IPS the cloud provider changes the HOST IP randomly to a IP which is available on node. With this behavior we get routing problems inside the Openshift-cluster.

Normal and expected state:

11:20 $ oc get hostsubnet
NAME                                 HOST                                 HOST IP       SUBNET           EGRESS CIDRS       EGRESS IPS
s-cp-infra-01.cloud.mycompany.com    s-cp-infra-01.cloud.mycompany.com    10.20.15.31   10.128.0.0/23    [10.20.15.96/28]   []
s-cp-infra-02.cloud.mycompany.com    s-cp-infra-02.cloud.mycompany.com    10.20.15.32   10.129.0.0/23    [10.20.15.96/28]   []
s-cp-infra-03.cloud.mycompany.com    s-cp-infra-03.cloud.mycompany.com    10.20.15.33   10.130.14.0/23   [10.20.15.96/28]   [10.20.15.102, 10.20.15.98, 10.20.15.99, 10.20.15.100, 10.20.15.97, 10.20.15.101, 10.20.15.103]
s-cp-master-01.cloud.mycompany.com   s-cp-master-01.cloud.mycompany.com   10.20.15.11   10.131.0.0/23    []                 []
...

s-cp-infra-03.cloud.mycompany.com changes the IP address nearly every second which one IPs available trough EGRESS IPS and the HOST IP


NAME                                 HOST                                 HOST IP       SUBNET           EGRESS CIDRS       EGRESS IPS
s-cp-infra-01.cloud.mycompany.com    s-cp-infra-01.cloud.mycompany.com    10.20.15.31   10.128.0.0/23    [10.20.15.96/28]   []
s-cp-infra-02.cloud.mycompany.com    s-cp-infra-02.cloud.mycompany.com    10.20.15.32   10.129.0.0/23    [10.20.15.96/28]   []
s-cp-infra-03.cloud.mycompany.com    s-cp-infra-03.cloud.mycompany.com    10.20.15.99   10.130.14.0/23   [10.20.15.96/28]   [10.20.15.102, 10.20.15.98, 10.20.15.99, 10.20.15.100, 10.20.15.97, 10.20.15.101, 10.20.15.103]
s-cp-master-01.cloud.mycompany.com   s-cp-master-01.cloud.mycompany.com   10.20.15.11   10.131.0.0/23    []                 []
..

What you expected to happen: host IP should be stable

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

Server https://s-openshift.mycompany.com:443 openshift v3.11.0+6c2b013-59 kubernetes v1.11.0+d4cacc0

- Cloud provider or hardware configuration:

[Global] user = "XXXXX@vsphere.local" password = "XXXXXXX" port = 443 insecure-flag = 1 datacenter = "OCP-Datacenter"

[VirtualCenter "10.4.4.102"]

[Workspace] server = "10.4.4.102" datacenter = "OCP-Datacenter" folder = "/OCP-Datacenter/vm/" default-datastore = "iscsi-hdd"

[Disk] scsicontrollertype = pvscsi

[Network] public-network = "n-vlan-openshift-staging-pub"

- OS (e.g. from /etc/os-release):

NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

- Kernel (e.g. `uname -a`):

Linux s-cp-master-01.cloud.mycompany.com 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux