search

cybertec-postgresql / vip-manager

Manages a virtual IP based on state kept in etcd or Consul
BSD 2-Clause "Simplified" License
207 stars 41 forks source link

The vip-manager did not raise the VIP address after the failover #247

Closed vitabaks closed 2 months ago

vitabaks commented 3 months ago

Please take a look at this problem. The vip-manager did not raise the VIP address after the failover.

We had a failure on 73-pg and 74-pg was successfully elected leader. However, the vip is currently not being advertised from 74-pg

Note: the actual value of the IP address has been masked

log:

root@server74-pg:~# journalctl -u vip-manager.service -n 10000 | grep -B10 -A10 -v "state is false, desired false"
Jul 25 09:12:57 server74-pg vip-manager[665]: 2024/07/25 09:12:57 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:13:07 server74-pg vip-manager[665]: 2024/07/25 09:13:07 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:13:17 server74-pg vip-manager[665]: 2024/07/25 09:13:17 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:13:27 server74-pg vip-manager[665]: 2024/07/25 09:13:27 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:13:37 server74-pg vip-manager[665]: 2024/07/25 09:13:37 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:13:47 server74-pg vip-manager[665]: 2024/07/25 09:13:47 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:13:57 server74-pg vip-manager[665]: 2024/07/25 09:13:57 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:14:07 server74-pg vip-manager[665]: 2024/07/25 09:14:07 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:14:17 server74-pg vip-manager[665]: 2024/07/25 09:14:17 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:14:27 server74-pg vip-manager[665]: 2024/07/25 09:14:27 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:14:29 server74-pg vip-manager[665]: 2024/07/25 09:14:29 Current Leader from DCS: server73-pg
Jul 25 09:14:37 server74-pg vip-manager[665]: 2024/07/25 09:14:37 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:14:47 server74-pg vip-manager[665]: 2024/07/25 09:14:47 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:14:57 server74-pg vip-manager[665]: 2024/07/25 09:14:57 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:15:07 server74-pg vip-manager[665]: 2024/07/25 09:15:07 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:15:17 server74-pg vip-manager[665]: 2024/07/25 09:15:17 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:15:27 server74-pg vip-manager[665]: 2024/07/25 09:15:27 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:15:37 server74-pg vip-manager[665]: 2024/07/25 09:15:37 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:15:47 server74-pg vip-manager[665]: 2024/07/25 09:15:47 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:15:57 server74-pg vip-manager[665]: 2024/07/25 09:15:57 IP address ***.***.***.82/32 state is false, desired false
Jul 25 09:16:07 server74-pg vip-manager[665]: 2024/07/25 09:16:07 IP address ***.***.***.82/32 state is false, desired false
root@server74-pg:~# 
root@server74-pg:~# journalctl -u vip-manager.service -f
Jul 26 12:39:47 server74-pg vip-manager[665]: 2024/07/26 12:39:47 IP address ***.***.***.82/32 state is false, desired false
Jul 26 12:39:57 server74-pg vip-manager[665]: 2024/07/26 12:39:57 IP address ***.***.***.82/32 state is false, desired false
Jul 26 12:40:07 server74-pg vip-manager[665]: 2024/07/26 12:40:07 IP address ***.***.***.82/32 state is false, desired false
Jul 26 12:40:17 server74-pg vip-manager[665]: 2024/07/26 12:40:17 IP address ***.***.***.82/32 state is false, desired false
Jul 26 12:40:27 server74-pg vip-manager[665]: 2024/07/26 12:40:27 IP address ***.***.***.82/32 state is false, desired false
Jul 26 12:40:37 server74-pg vip-manager[665]: 2024/07/26 12:40:37 IP address ***.***.***.82/32 state is false, desired false
Jul 26 12:40:47 server74-pg vip-manager[665]: 2024/07/26 12:40:47 IP address ***.***.***.82/32 state is false, desired false
Jul 26 12:40:57 server74-pg vip-manager[665]: 2024/07/26 12:40:57 IP address ***.***.***.82/32 state is false, desired false
Jul 26 12:41:07 server74-pg vip-manager[665]: 2024/07/26 12:41:07 IP address ***.***.***.82/32 state is false, desired false
Jul 26 12:41:17 server74-pg vip-manager[665]: 2024/07/26 12:41:17 IP address ***.***.***.82/32 state is false, desired false
^C
root@server74-pg:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP group default qlen 1000
    link/ether 52:54:00:ea:0b:f7 brd ff:ff:ff:ff:ff:ff
    inet ***.***.***.228/27 brd 216.223.147.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:feea:bf7/64 scope link 
       valid_lft forever preferred_lft forever
root@server74-pg:~# 
root@server74-pg:~# grep -v '^ *#\|^$' /etc/patroni/vip-manager.yml 
interval: 1000
trigger-key: "/service/postgres-cluster/leader"
trigger-value: "server74-pg"
ip: ***.***.***.82 # the virtual ip address to manage
netmask: 32 # netmask for the virtual ip
interface: enp1s0 # interface to which the virtual ip will be added
hosting-type: basic # possible values: basic, or hetzner.
dcs-type: etcd # etcd or consul
dcs-endpoints:
  - http://216.***.***.53:2379
  - http://216.***.***.103:2379
  - http://216.***.***.231:2379
retry-num: 2
retry-after: 250  #in milliseconds
verbose: false
root@server74-pg:~# vip-manager --version
version: 2.5.0

root@server74-pg:~# curl -L http://216.***.***53:2379/version
{"etcdserver":"3.5.13","etcdcluster":"3.5.0"}

Patroni log

root@server74-pg:~# journalctl -u patroni -f 
Jul 26 12:53:00 server74-pg patroni[699]: 2024-07-26 12:53:00,219 INFO: no action. I am (server74-pg), the leader with the lock
Jul 26 12:53:10 server74-pg patroni[699]: 2024-07-26 12:53:10,207 INFO: no action. I am (server74-pg), the leader with the lock
Jul 26 12:53:20 server74-pg patroni[699]: 2024-07-26 12:53:20,680 INFO: no action. I am (server74-pg), the leader with the lock
Jul 26 12:53:30 server74-pg patroni[699]: 2024-07-26 12:53:30,101 INFO: no action. I am (server74-pg), the leader with the lock
Jul 26 12:53:40 server74-pg patroni[699]: 2024-07-26 12:53:40,227 INFO: no action. I am (server74-pg), the leader with the lock
Jul 26 12:53:50 server74-pg patroni[699]: 2024-07-26 12:53:50,229 INFO: no action. I am (server74-pg), the leader with the lock
Jul 26 12:54:00 server74-pg patroni[699]: 2024-07-26 12:54:00,198 INFO: no action. I am (server74-pg), the leader with the lock
Jul 26 12:54:10 server74-pg patroni[699]: 2024-07-26 12:54:10,150 INFO: no action. I am (server74-pg), the leader with the lock
Jul 26 12:54:20 server74-pg patroni[699]: 2024-07-26 12:54:20,158 INFO: no action. I am (server74-pg), the leader with the lock
Jul 26 12:54:30 server74-pg patroni[699]: 2024-07-26 12:54:30,291 INFO: no action. I am (server74-pg), the leader with the lock
vitabaks commented 3 months ago

@pashagolub I have upgraded to version 2.6.0

root@server74-pg:~# systemctl restart vip-manager
root@server74-pg:~# 
root@server74-pg:~# journalctl -u vip-manager.service -f
Jul 26 13:05:42 server74-pg vip-manager[445785]: 2024/07/26 13:05:42 current leader from DCS: server74-pg
Jul 26 13:05:42 server74-pg vip-manager[445785]: 2024/07/26 13:05:42 IP address ***.***.***.82/32 state is false, desired true
Jul 26 13:05:42 server74-pg vip-manager[445785]: 2024/07/26 13:05:42 Configuring address ***.***.***.82/32 on enp1s0
Jul 26 13:05:42 server74-pg vip-manager[445785]: 2024/07/26 13:05:42 Sent gratuitous ARP reply
Jul 26 13:05:42 server74-pg vip-manager[445785]: 2024/07/26 13:05:42 Sent gratuitous ARP request
Jul 26 13:05:42 server74-pg vip-manager[445785]: 2024/07/26 13:05:42 IP address ***.***.***.82/32 state is true, desired true
Jul 26 13:05:52 server74-pg vip-manager[445785]: 2024/07/26 13:05:52 IP address ***.***.***.82/32 state is true, desired true
Jul 26 13:06:02 server74-pg vip-manager[445785]: 2024/07/26 13:06:02 IP address ***.***.***.82/32 state is true, desired true
Jul 26 13:06:12 server74-pg vip-manager[445785]: 2024/07/26 13:06:12 IP address ***.***.***.82/32 state is true, desired true
Jul 26 13:06:22 server74-pg vip-manager[445785]: 2024/07/26 13:06:22 IP address ***.***.***.82/32 state is true, desired true
^C

root@server74-pg:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP group default qlen 1000
    link/ether 52:54:00:ea:0b:f7 brd ff:ff:ff:ff:ff:ff
    inet ***.***.***.228/27 brd 216.223.147.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet ***.***.***.82/32 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:feea:bf7/64 scope link 
       valid_lft forever preferred_lft forever
root@server74-pg:~# 

root@server74-pg:~# vip-manager --version
version: 2.6.0

I do not know what helped, just restarting the service or updating. We'll observe.

ilapcom commented 3 months ago

@vitabaks - vip-manager version 2.6 correctly adding | removing vip when switching between 72-pg | 73-pg | 74-pg

vitabaks commented 3 months ago

Thanks @ilapcom