pqarmitage
commented
4 years ago There have been some issue with keepalived and handling termination of scripts that were resolved in v2.0.20, so can you please build and run that version and see if you still get the problem.
With the configuration you have, once keepalived has started running it is doing nothing other that running the scripts. Since the scripts have a timeout of 15 seconds and a retry time of 2 seconds, keepalived really should not be busy. This suggests that there is some problem with running the scripts.
Are the failed to terminate after kill messages being produced whil keepalived is using 100% cpu time?
I think what you will need to do when/if the problem happens again after upgrading to v2.0.20 is to 1) Attach to the process that is 100% with gdb and generate a stack backtrace. Please do this several times so that we can try and see what functions keepalived is looping in 2) Try running strace again to see if it produces any output
If the problem still persists then it will be necessary to do a debug build of keepalived so that we can get a thread dump whill keepalived is looping.
szpajder
fliespl
dfoxg
midorinet
Describe the bug After the restart of the process, within 1-5 days, the keepalived process begins to use 100 percent of the CPU in Userspace
To Reproduce systemctl restart keepalived wait 2-5 day
Keepalived version keepalived-2.0.10-4.el8_0.2.x86_64 Output of
keepalived -vKeepalived v2.0.10 (11/12,2018)Copyright(C) 2001-2018 Alexandre Cassen, acassen@gmail.com
Built with kernel headers for Linux 4.18.0 Running on Linux 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Tue Sep 24 11:32:19 UTC 2019
configure options: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-snmp --enable-snmp-rfc --enable-sha1 --with-init=systemd build_alias=x86_64-redhat-linux-gnu host_alias=x86_64-redhat-linux-gnu PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection LDFLAGS=-Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
Config options: LIBIPSET_DYNAMIC LVS VRRP VRRP_AUTH OLD_CHKSUM_COMPAT FIB_ROUTING SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3
System options: PIPE2 SIGNALFD INOTIFY_INIT1 VSYSLOG EPOLL_CREATE1 IPV4_DEVCONF LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_OIFNAME FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS IP_MULTICAST_ALL LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA LIBIPTC LIBIPSET_PRE_V7 LIBIPVS_NETLINK IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS VRRP_VMAC SOCK_NONBLOCK SOCK_CLOEXEC O_PATH GLOB_BRACE INET6_ADDR_GEN_MODE VRF SO_MARK SCHED_RT SCHED_RESET_ON_FORK Distro (please complete the following information): NAME="CentOS Linux" VERSION="8 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="8" PLATFORM_ID="platform:el8" PRETTY_NAME="CentOS Linux 8 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:8" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8" CENTOS_MANTISBT_PROJECT_VERSION="8" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="8" Details of any containerisation or hosted service (e.g. AWS) If keepalived is being run in a container or on a hosted service, provide full details
Configuration file: ! Configuration File for keepalived
! configure for sflow virtual_server 1.1.1.1 6343 { delay_loop 6 lb_algo sh lb_kind NAT protocol UDP real_server 2.2.2.2 6343 { weight 1 MISC_CHECK { misc_path "/opt/lb.sh 2.2.2.2” retry 2 misc_timeout 15 } } real_server 3.3.3.3 6343 { weight 1 MISC_CHECK { misc_path "/opt/lb.sh 3.3.3.3” retry 2 misc_timeout 15 } }
real_server 4.4.4.4 6343 { weight 1 MISC_CHECK { misc_path "/opt/lb.sh 4.4.4.4” retry 2 misc_timeout 15 } } real_server 5.5.5.5 6343 { weight 1 MISC_CHECK { misc_path "/opt/lb.sh 5.5.5.5” retry 2 misc_timeout 15 } } } Notify and track scripts cat /opt/lb.sh
!/bin/bash
IP=$1
report=$(/usr/bin/curl -s --connect-timeout 3 --insecure --user user:password https://"${IP}"/api/gateway/1.5/system/status | grep 'sensor|diskmon|memmonitor' | grep Running | wc -l)
echo $report
if [ "$report" -eq "3" ]; then exit 0 else exit 1 fi
System Log entries Full keepalived system log entries from when keepalived started
Did keepalived coredump? debug with the help of the strace utility does not show anything on a process that is already 100% loaded
Additional context
root@host1~]# ps aux | grep keepalived root 1883 0.0 0.0 9184 1084 pts/0 R+ 12:20 0:00 grep --color=auto keepalived root 29456 0.0 0.0 111520 1012 ? Ss 2019 0:00 /usr/sbin/keepalived -D root 29457 91.8 0.1 124208 6180 ? R 2019 66275:07 /usr/sbin/keepalived -D
[root@host1~]# curl --version curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.1.1 zlib/1.2.11 brotli/1.0.6 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh/0.8.5/openssl/zlib nghttp2/1.33.0 Release-Date: 2018-09-05 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink
sysctl.conf
net.core.rmem_max=26214400 net.core.rmem_default=26214400 net.ipv4.ip_forward=1 net.ipv4.vs.expire_nodest_conn=1 net.ipv4.vs.expire_quiescent_template=1
[root@ipvs-host1t ~]# systemctl status keepalived ● keepalived.service - LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2019-12-12 10:09:41 MSK; 1 months 19 days ago Process: 29455 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 29456 (keepalived) Tasks: 2 (limit: 24896) Memory: 10.9M CGroup: /system.slice/keepalived.service ├─29456 /usr/sbin/keepalived -D └─29457 /usr/sbin/keepalived -D
Dec 16 08:22:06 host1 Keepalived_healthcheckers[29457]: Child (PID 79470) failed to terminate after kill Dec 16 08:22:10 host1 Keepalived_healthcheckers[29457]: Child (PID 83476) failed to terminate after kill Dec 16 08:22:13 host1 Keepalived_healthcheckers[29457]: Child (PID 83488) failed to terminate after kill Dec 16 08:22:14 host1 Keepalived_healthcheckers[29457]: Child (PID 83452) failed to terminate after kill Dec 16 08:22:14 host1Keepalived_healthcheckers[29457]: Child (PID 83494) failed to terminate after kill Dec 16 08:22:16 host1 Keepalived_healthcheckers[29457]: Child (PID 79470) failed to terminate after kill Dec 16 08:22:20 host1 Keepalived_healthcheckers[29457]: Child (PID 83476) failed to terminate after kill Dec 16 08:22:23 host1 Keepalived_healthcheckers[29457]: Child (PID 83488) failed to terminate after kill Dec 16 08:22:24 host1 Keepalived_healthcheckers[29457]: Child (PID 83452) failed to terminate after kill Dec 16 08:22:24 host1 Keepalived_healthcheckers[29457]: Child (PID 83494) failed to terminate after kill [root@ihost1 ~]# ps aux | grep keepalived root 1883 0.0 0.0 9184 1084 pts/0 R+ 12:20 0:00 grep --color=auto keepalived root 29456 0.0 0.0 111520 1012 ? Ss 2019 0:00 /usr/sbin/keepalived -D root 29457 91.8 0.1 124208 6180 ? R 2019 66275:07 /usr/sbin/keepalived -D [root@host1 ~]# curl -v curl: no URL specified! curl: try 'curl --help' or 'curl --manual' for more information [root@host1 ~]# curl --version curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.1.1 zlib/1.2.11 brotli/1.0.6 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh/0.8.5/openssl/zlib nghttp2/1.33.0 Release-Date: 2018-09-05 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink