search

bingooyong / note

1 stars 1 forks source link

玩转 Redis 高可用方案 #11

Open bingooyong opened 5 years ago

bingooyong commented 5 years ago

Redis 高可用方案

bingooyong commented 5 years ago

Redis 高可用方案

[TOC]

Redis的高可用方案有很多种，都有自己的优缺点，我们需要根据具体场景进行选择最适合自己的。

  1. Redis Master-Slave + Keepalived + VIP 这是很经典的db架构，也可以用与mysql的主从切换

    基本原理是：Keepalive通过脚本检测master的存活，然后通过漂移VIP（Virtual IP）完成主从切换。

  2. Redis Master-Slave + Sentinel+ DNS Service 基本原理是Sentinel集群进行Redis的存活检测和Redis M-S状态切换。完成切换之后，sentinel调用notification-script参数制定的配置文件，通知DNS Server更改DNS配置，master dns解析执行新的master。

  3. Redis Master-Slave + Sentinel + Configure Center（Zookeeper） 基本原理和第三种方案相似，只是notification-script通知的是配置中心完成redis连接配置的修改，比如Zookeeper实现的配置中心。

  4. Redis Master-Slave + Sentinel + Twemproxy + Lvs 这种方案层次比较多，sentinel通知twemproxy进行redis m-s的配置更改

  5. Redis Master-Slave + Sentinel + Keepalived + Haproxy

  6. Redis Cluster

    1 Redis Master-Slave + Keepalived + VIP

  7. 基本构建与原理

1）Keepalived + VIP : 在redis master-slave上部署keepalived、redis instance存活检测脚本、以及告警通知脚本。

2）当redis master失效的时候，VIP从master上漂移到slave上，完成m-s角色和配置更改。3）客户端连接redis的参数中host设置的是VIP，整个切换过程对客户端透明。

  1. 优缺点与适用场景

优点：实现简单，成本低，整个切换过程对客户端透明。

缺点：整个集群的最大吞吐量受限于redis单实例的处理能力，除非一个应用使用多套这种Keepalived+VIP方案。因而扩展能力较差，而且不适合目前单机部署多个redis实例的部署场景。

适合场景：并发请求不是很高的应用。

1.1 安装 keepalived 和 Redis

1.1.1 安装 keepalived

keepalived

  1. 使用yum源安装

    yum install -y keepalived

  2. 使用源码安装

    yum -y install openssl-devel libnl3-devel ipset-devel iptables-devel libnfnetlink-devel net-snmp-devel

wget -P /usr/local/src/ https://www.keepalived.org/software/keepalived-1.3.5.tar.gz
cd /usr/local/src/
tar zxf  keepalived-1.3.5.tar.gz
cd keepalived-1.3.5
./configure --prefix=/usr/local/keepalived/ 
make && make install

cp /usr/local/src/keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/init.d/keepalived 
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/ 
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ 
mkdir -p /etc/keepalived/ 
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf 
# 
/etc/keepalived/keepalived.conf 是默认的配置文件

    1.1.2 安装 Redis

下载安装 redis

wget -P /opt/soft http://download.redis.io/releases/redis-4.0.9.tar.gz
cd /opt/soft && tar xzf redis-4.0.9.tar.gz
cd redis-4.0.9 && make PREFIX=/opt/redis install
mkdir -p /var/run /var/log /var/lib/redis/6379
cd /opt/redis && touch redis.conf

修改配置文件 redis.conf 

daemonize yes
pidfile /var/run/redis_6379.pid
port 6379
tcp-backlog 511
bind 0.0.0.0
timeout 0
tcp-keepalive 0
loglevel notice
logfile /var/log/redis_6379.log
databases 16
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis/6379
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
maxclients 10000
maxmemory 17179869184
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

启动

/opt/redis/bin/redis-server /opt/redis/redis.conf

1.2 主从部署 keepalived + Redis

1.2.1 部署拓补图

IP地址 角色
192.168.33.70 Master
192.168.33.80 Slave
192.168.33.90 VIP 

                            +----------+
                            |          |
                            |  uplink  |
                            |          |
                            +-----+----+
                                  |
                                  |
                                  |
                                  |
                                  |
   MASTER                   keepalived vip                BACKUP
192.168.33.70               192.168.33.90              192.168.33.80
+------------+             +------+------+            +------------+
|            |             |             |            |            |
|   redis    +-------------+ virtual vip +------------+   redis    |
|            |             |             |            |            |
+------------+             +-------------+            +------------+
    MASTER                                               SLAVE

1.2.2 Redis主从配置

无需调整，使用keepalived来切换主从

1.2.3 配置 keepalived scripts for redis

所有主机脚本都一致，配置Redis的相关脚本

1.2.3.1 redis_check.sh

vim /etc/keepalived/scripts/redis_check.sh 

#!/bin/bash
###/etc/keepalived/scripts/redis_check.sh

ALIVE=`/usr/local/bin/redis-cli PING`
LOGFILE="/var/log/keepalived/keepalived-redis-check.log"
pid=$$

for ((i=0; i<2; i++))
do
    if [ "$ALIVE" == "PONG" ]; then
        echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[check] Success: PING $ALIVE " >> $LOGFILE 2>&1
        exit 0
    else
        echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[check] Failed: PING $ALIVE " >> $LOGFILE 2>&1
        sleep 1
    fi
done
exit 1
1.2.3.2 redis_master.sh

vim /etc/keepalived/scripts/redis_master.sh 

#!/bin/bash
###/etc/keepalived/scripts/redis_master.sh

REDISCLI="/usr/local/bin/redis-cli"
LOGFILE="/var/log/keepalived/keepalived-redis-state.log"
pid=$$
host=$1
port=$2

echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[slaver]" >> $LOGFILE 2>&1
echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[slaver] Run 'SLAVEOF $host $port'" >> $LOGFILE 2>&1
$REDISCLI SLAVEOF $host $port>> $LOGFILE  2>&1
echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[slaver] wait 10 sec for data sync from old master" >> $LOGFILE 2>&1
sleep 10
echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[slaver] data rsync from old mater ok..." >> $LOGFILE 2>&1
echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[master] Run slaveof no one,close master/slave" >> $LOGFILE 2>&1
$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1
echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[master] wait other slave connect...." >> $LOGFILE 2>&1
1.2.3.3 redis_backup.sh

vim /etc/keepalived/scripts/redis_backup.sh 

#!/bin/bash
###/etc/keepalived/scripts/redis_backup.sh

REDISCLI="/usr/local/bin/redis-cli"
LOGFILE="/var/log/keepalived/keepalived-redis-state.log"
pid=$$
host=$1
port=$2

echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[master] Being slave state..." >> $LOGFILE 2>&1
echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[master] wait 10 sec for data sync from old master" >> $LOGFILE 2>&1
sleep 10
echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[master] data rsync from old mater ok..." >> $LOGFILE 2>&1
echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[slaver] Run 'SLAVEOF $host $port'" >> $LOGFILE 2>&1
$REDISCLI SLAVEOF $host $port >> $LOGFILE  2>&1
echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[slaver] slave connect to $host ok..." >> $LOGFILE 2>&1
1.2.3.4 redis_fault.sh

vim /etc/keepalived/scripts/redis_fault.sh 

#!/bin/bash
###/etc/keepalived/scripts/redis_fault.sh

LOGFILE="/var/log/keepalived/keepalived-redis-state.log"
pid=$$

echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[fault]" >> $LOGFILE 2>&1
1.2.3.5 redis_stop.sh

vim /etc/keepalived/scripts/redis_stop.sh 

#!/bin/bash
###/etc/keepalived/scripts/redis_stop.sh

LOGFILE="/var/log/keepalived/keepalived-redis-state.log"
pid=$$

echo "`date +'%Y-%m-%d %H:%M:%S'`|$pid|state:[stop]" >> $LOGFILE 2>&1

1.2.4 配置 keepalived.conf

192.168.33.70 keepalived.master

global_defs {
    lvs_id LVS_redis
}

vrrp_script chk_redis {
    script "/etc/keepalived/scripts/redis_check.sh"
    interval 2
    timeout 2
    fall 3
}

vrrp_instance rk {
    state MASTER
    interface enp0s8
    virtual_router_id 54
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 6ea5d4e2
    }
    track_script {
        chk_redis
    }
    virtual_ipaddress {
       192.168.33.90/24
    }
    notify_master "/etc/keepalived/scripts/redis_master.sh 192.168.33.90 6379"
    notify_backup "/etc/keepalived/scripts/redis_backup.sh 192.168.33.80 6379"
    notify_fault  /etc/keepalived/scripts/redis_fault.sh
    notify_stop   /etc/keepalived/scripts/redis_stop.sh
}

192.168.33.80 keepalived.slave

global_defs {
    lvs_id LVS_redis
}

vrrp_script chk_redis {
    script "/etc/keepalived/scripts/redis_check.sh"
    interval 2
    timeout 2
    fall 3
}

vrrp_instance rk {
    state BACKUP
    interface enp0s8
    virtual_router_id 54
    priority 98
    advert_int  1
    authentication {
        auth_type PASS
        auth_pass 6ea5d4e2
    }
    track_script {
        chk_redis
    }
    virtual_ipaddress {
        192.168.33.90/24
    }
    notify_master "/etc/keepalived/scripts/redis_master.sh 192.168.33.90 6379"
    notify_backup "/etc/keepalived/scripts/redis_backup.sh 192.168.33.70 6379"
    notify_fault  /etc/keepalived/scripts/redis_fault.sh
    notify_stop   /etc/keepalived/scripts/redis_stop.sh
}

1.3 验证

# 启动 keepalived
service keepalived start

# 启动 redis
/opt/redis/bin/redis-server /opt/redis/redis.conf

分别在不同机器查看VIP情况命令 ip a

192.168.33.70 机器有VIP信息 192.168.33.90

3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:a8:a3:25 brd ff:ff:ff:ff:ff:ff
    inet 192.168.33.70/24 brd 192.168.33.255 scope global enp0s8
       valid_lft forever preferred_lft forever
    inet 192.168.33.90/24 scope global secondary enp0s8
       valid_lft forever preferred_lft forever

192.168.33.80 机器没有VIP信息

3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:b7:a4:e2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.33.80/24 brd 192.168.33.255 scope global enp0s8
       valid_lft forever preferred_lft forever

查看当前集群Redis状态

➜  ~ redis-cli -h 192.168.33.90 -p 6379 info replication | grep "role"
role:master
➜  ~ redis-cli -h 192.168.33.70 -p 6379 info replication | grep "role"
role:master
➜  ~ redis-cli -h 192.168.33.80 -p 6379 info replication | grep "role"
role:slave

停止 192.168.33.70上的Redis服务，等待10秒后查看集群状态

pkill redis-server

集群的 Master 已经切换到 192.168.33.80 上了，并且VIP可以正常工作

➜  ~ redis-cli -h 192.168.33.90 -p 6379 info replication | grep "role"
role:master
➜  ~ redis-cli -h 192.168.33.70 -p 6379 info replication | grep "role"
Could not connect to Redis at 192.168.33.70:6379: Connection refused
➜  ~ redis-cli -h 192.168.33.80 -p 6379 info replication | grep "role"
role:master

分别在不同机器查看VIP情况命令 ip a

192.168.33.70 机器没有VIP信息

3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:a8:a3:25 brd ff:ff:ff:ff:ff:ff
    inet 192.168.33.70/24 brd 192.168.33.255 scope global enp0s8
       valid_lft forever preferred_lft forever

192.168.33.80 机器有VIP信息 192.168.33.90

3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:b7:a4:e2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.33.80/24 brd 192.168.33.255 scope global enp0s8
       valid_lft forever preferred_lft forever
    inet 192.168.33.90/24 scope global secondary enp0s8
       valid_lft forever preferred_lft forever

在 VIP 上写一个测试key，待192.168.33.70恢复后，检查数据是否同步

redis-cli -h 192.168.33.90 -p 6379 set testkey testval

在 192.168.33.70 上启动Redis，等待10秒查看集群状态，期望接管VIP，并能成功获取测试key的值

/opt/redis/bin/redis-server /opt/redis/redis.conf

查看测试key的值

➜  ~ redis-cli -h 192.168.33.70 -p 6379 get testkey
"testval"
➜  ~ redis-cli -h 192.168.33.80 -p 6379 get testkey
"testval"
➜  ~ redis-cli -h 192.168.33.90 -p 6379 get testkey
"testval"

1.4 注意事项

关闭SELinux，权限配置太复杂，会导致keepalived执行脚本失败等原因

查看SELinux状态， 如果SELinux status参数为enabled即为开启状态

➜  ~ /usr/sbin/sestatus -v
SELinux status:                 disabled

getenforce，也可以用这个命令检查

➜  ~ getenforce
Disabled

关闭SELinux：

1、临时关闭（不用重启机器）：

#关闭：设置SELinux 成为permissive模式
setenforce 0

#开启：设置SELinux 成为enforcing模式
setenforce 1

2、修改配置文件需要重启机器：

修改/etc/selinux/config 文件

将SELINUX=enforcing改为SELINUX=disabled

重启机器即可