Kubernetes master node HA （keepalived+haproxy）

[zhengbo0]

运行keepalived镜像后，出现异常

IPVS: Can't initialize ipvs: Protocol not available

尝试在Dockefile里面加入

apt install ipvsadm

仍然不行

[zhengbo0]

由于keepalived需要通过IPVS模块实现路由转发,所以我们需要​使能内核IPVS模块。 从Linux内核版本2.6起,ip_vs code已经被整合进了内核中,因此,只要在编译内核的时候选 择了ipvs的功能,Linux即能支持LVS。因此我们只需要配置操作系统启动时自动加载IPVS模 块（在物理机上执行）:

echo "ip_vs" >> /etc/modules 
echo "ip_vs_rr" >> /etc/modules 
echo "ip_vs_wrr" >> /etc/modules 

我们可以通过如下命令查看ip_vs模块是否成功加载: lsmod | grep ip_vs 如果没有加载,我们可以通过modprobe命令加载该模块:

modprobe ip_vs 
modprobe ip_vs_rr 
modprobe ip_vs_wrr 

为了使keepalived将数据包转发到真实的后端服务器,每一个lb node都需要开启IP转发功能​ echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf 另外,keepalived设置的VIP有可能为非本地IP地址,所以我们还需要使能非本地IP地址绑定功能 echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf

[zhengbo0]

发现出现新的异常

IPVS: Can't initialize ipvs: Permission denied (you must be root)

尝试注意掉keepalived.conf中后异常消失，说明问题是这个健康检查脚本导致的

vrrp_script chk_http_port {
    #script "/opt/check_haproxy.sh"
    interval 2
    weight 2
}

[zhengbo0]

实际上已经是root权限了，在Dockerfile中添加 USER: root未解决 发现是Docker导致的，需要在docker的时候加入 --privileged=true问题解决 或者在pod定义的yaml文件中添加

securityContext:
        privileged: true

[zhengbo0]

查看pod 中keepalived容器日志，发现新的异常 Default interface eth0 does not exist and no interface specified. Skip VRRP address. 查看worker节点网卡名称不是eth0,修改keepalived.conf中eth0为eno1

vrrp_instance VI_1 {
    state MASTER       
    interface eno1
    virtual_router_id 51
    priority 100 
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass bfd
    }
    track_script { 
        chk_http_port
    }
    virtual_ipaddress {
       10.1.0.200
    }
}

[zhengbo0]

重新启动后出现新的错误

VRRP_Instance(VI_1) ignoring received advertisment...
ip address associated with VRID not present in received packet : 10.1.0.200
one or more VIP associated with VRID mismatch actual MASTER advert
bogus VRRP packet received on eno1 !!!

[zhengbo0]

haproxy代理ssl配置有两种方式: 1) haproxy本身提供SSL证书,后面的web服务器走正常的http协议; 2) haproxy本身只提供代理,直接转发client端的HTTPS请求到后端的web服务器。注意:这 种模式下“mode”必须是“tcp”模式, 即仅支持4层代理。 考虑到需要7层代理的支持,采用了第一种方式 参考：https://serversforhackers.com/using-ssl-certificates-with-haproxy 配置haproxy ssl,运行容器时出现异常：

<7>haproxy-systemd-wrapper: executing /usr/local/sbin/haproxy -p /run/haproxy.pid -f /usr/local/etc/haproxy/haproxy.cfg -Ds 
[ALERT] 216/130456 (9) : parsing [/usr/local/etc/haproxy/haproxy.cfg:9] : unknown keyword 'tune.ssl.default­dh­param' in 'global' section
[ALERT] 216/130456 (9) : parsing [/usr/local/etc/haproxy/haproxy.cfg:32]: unknown stats parameter 'show­node', expects 'admin', 'uri', 'realm', 'auth', 'scope', 'enable', 'hide-version', 'show-node', 'show-desc' or 'show-legends'.
[ALERT] 216/130456 (9) : parsing [/usr/local/etc/haproxy/haproxy.cfg:48] : unknown option 'forward­for'.
[ALERT] 216/130456 (9) : Error(s) found in configuration file : /usr/local/etc/haproxy/haproxy.cfg
[WARNING] 216/130456 (9) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
[ALERT] 216/130456 (9) : Fatal errors found in configuration.
<5>haproxy-systemd-wrapper: exit, haproxy RC=256

执行命令“haproxy ­f /etc/haproxy/haproxy.cfg”时,真正执行的是: “/usr/local/sbin/haproxy ­p /run/haproxy.pid ­f /etc/haproxy/haproxy.cfg ­Ds”,对于“­Ds”选项, 官网是这么描述的:

­Ds passe en daemon systemd This patch adds a new option "­Ds" which is exactly like "­D", but instead of forking n times to get n jobs running and then exiting, prefers to wait for all the children it just created. With this done, haproxy becomes more systemd­compliant, without changing anything for other systems.

原来,“haproxy”经过“haproxy­systemd­wrapper”包装后在后台执行,而docker container不允 许进程后台执行,否则docker容器将该启动命令执行完后就退出了。 修改Dockfile中haproxy的启动方式

CMD ["haproxy", "-f", "/usr/local/etc/haproxy/haproxy.cfg"]

修改为

CMD ["haproxy", "-f", "/usr/local/etc/haproxy/haproxy.cfg","-p","/var/run/haproxy.pid"]