search

w13915984028 / harvester-develop-summary

Summary of Harvester develop.
2 stars 4 forks source link

Reproduce of https://github.com/harvester/harvester/issues/1668 #1

Open w13915984028 opened 2 years ago

w13915984028 commented 2 years ago

Reproduce the issue: https://github.com/harvester/harvester/issues/1668

  1. Install a new Harvester from a new VM in the same node image

two IPs are allocated from KVM embedded dhcp server, 192.168.122.222 is node IP, 192.168.222.97 is VIP

  1. Harvester installation image

  2. Harvester node reboot after installation

Reboot Harvester after installation
Node name changed, the node IP also changed

image

  1. In Harvester setting up, it uses node IP first image
virsh # net-dhcp-leases --network default
 Expiry Time           MAC address         Protocol   IP address           Hostname       Client ID or DUID
----------------------------------------------------------------------------------------------------------------------------------------------------
 2021-12-14 09:31:54   52:54:00:10:4f:08   ipv4       192.168.122.85/24    ubuntuvmday0   ff:56:50:4d:98:00:02:00:00:ab:11:b5:85:b2:62:9e:64:3c:a1
 2021-12-14 09:34:06   52:54:00:90:e2:26   ipv4       192.168.122.11/24    harvmain0112   ff:00:90:e2:26:00:01:00:01:29:3a:6b:a3:52:54:00:90:e2:26

 2021-12-14 09:22:47   52:54:00:ae:e9:e2   ipv4       192.168.122.222/24   rancher        ff:00:ae:e9:e2:00:01:00:01:29:4b:57:16:52:54:00:ae:e9:e2
 2021-12-14 09:34:24   52:54:00:ae:e9:e2   ipv4       192.168.122.223/24   harv1412       ff:00:ae:e9:e2:00:01:00:01:29:4b:59:dd:52:54:00:ae:e9:e2
 2021-12-14 09:24:45   6a:10:d2:c2:6a:3b   ipv4       192.168.122.97/24    -              -

virsh #

virsh # domifaddr harv1412 --full
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet56     52:54:00:ae:e9:e2    ipv4         192.168.122.222/24
 vnet56     52:54:00:ae:e9:e2    ipv4         192.168.122.223/24

virsh #

  1. Harvester VIP is ready now: image

  2. VIP is newed image

w13915984028 commented 2 years ago

**A small tricky:

VIP is allocated from KVM using MAC6a:10:d2:c2:6a:3b of vip-7e465ab1@harvester-mgmt, but it is bindded to harvester-mgmt**


rancher@harv1412:~> ip link

2: ens3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master harvester-mgmt state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:ae:e9:e2 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: harvester-mgmt: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:ae:e9:e2 brd ff:ff:ff:ff:ff:ff

65: vip-7e465ab1@harvester-mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 6a:10:d2:c2:6a:3b brd ff:ff:ff:ff:ff:ff

rancher@harv1412:~> ip addr

2: ens3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master harvester-mgmt state UP group default qlen 1000
    link/ether 52:54:00:ae:e9:e2 brd ff:ff:ff:ff:ff:ff
    altname enp0s3

3: harvester-mgmt: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:ae:e9:e2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.223/24 brd 192.168.122.255 scope global harvester-mgmt
       valid_lft forever preferred_lft forever
    inet 192.168.122.97/32 scope global harvester-mgmt
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:feae:e9e2/64 scope link
       valid_lft forever preferred_lft forever

65: vip-7e465ab1@harvester-mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 6a:10:d2:c2:6a:3b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6810:d2ff:fec2:6a3b/64 scope link
       valid_lft forever preferred_lft forever

When check from another VM, the Harvester node IP, VIP has same MAC 52:54:00:ae:e9:e2 (of node VM)

—- ubuntu VM —-

rancher@ubuntuvmday0:~$ ping 192.168.122.97
PING 192.168.122.97 (192.168.122.97) 56(84) bytes of data.
64 bytes from 192.168.122.97: icmp_seq=1 ttl=64 time=1.02 ms
64 bytes from 192.168.122.97: icmp_seq=2 ttl=64 time=0.495 ms

rancher@ubuntuvmday0:~$ ip neigh

192.168.122.97 dev enp1s0 lladdr 52:54:00:ae:e9:e2 REACHABLE
rancher@ubuntuvmday0:~$

rancher@ubuntuvmday0:~$ ping 192.168.122.223
PING 192.168.122.223 (192.168.122.223) 56(84) bytes of data.
64 bytes from 192.168.122.223: icmp_seq=1 ttl=64 time=0.492 ms

rancher@ubuntuvmday0:~$ ip neigh

192.168.122.223 dev enp1s0 lladdr 52:54:00:ae:e9:e2 DELAY

192.168.122.97 dev enp1s0 lladdr 52:54:00:ae:e9:e2 STALE
w13915984028 commented 2 years ago

The VIP is not renewed from KVM DHCP server, it is timed out.

VIP is still pingable, maybe it is because the MAC address is in HOST OS table, but when the MAC is also aged, maybe the VIP is finally lost.

image

image

image

w13915984028 commented 2 years ago

The VIP is seeing aged from KVM DHCP list.

The VIP is allocated from KVM using MAC 6a:10:d2:c2:6a:3b of vip-7e465ab1@harvester-mgmt, but it is attached to harvester-mgmt

It looks, when the Harvester goes into "ready" state, the VIP is last re-leased from DHCP server, after that, no renewing.

Which module is responsible for renewing the VIP? If it try to get VIP (ipv4) from dev vip-7e465ab1@harvester-mgmt, then it will fail, the VIP is attached to dev harvester-mgmt.

w13915984028 commented 2 years ago

consider enhancement https://github.com/harvester/harvester/issues/1683

w13915984028 commented 2 years ago

Even when keep the hostname as "rancher", it looks the NODE IP from KVM still changed, needs further investigation

image

w13915984028 commented 2 years ago

Even when keep the hostname as "rancher", it looks the NODE IP from KVM still changed, needs further investigation

The "Client ID or DUID" has a change, we did not notice it before

ff:00:f7:72:bf:00:01:00:01:29:4e_:04:bd_:52:54:00:f7:72:bf

ff:00:f7:72:bf:00:01:00:01:29:4e:_06:ff_:52:54:00:f7:72:bf
virsh # net-dhcp-leases --network default
 Expiry Time           MAC address         Protocol   IP address           Hostname       Client ID or DUID
----------------------------------------------------------------------------------------------------------------------------------------------------
 2021-12-16 10:23:11   46:6c:14:89:e8:ee   ipv4       192.168.122.225/24   -              -

 2021-12-16 10:08:14   52:54:00:f7:72:bf   ipv4       192.168.122.71/24    rancher        ff:00:f7:72:bf:00:01:00:01:29:4e:04:bd:52:54:00:f7:72:bf

 2021-12-16 10:17:38   52:54:00:f7:72:bf   ipv4       192.168.122.72/24    rancher        ff:00:f7:72:bf:00:01:00:01:29:4e:06