search

zshuangyan / blog

我的个人博客
2 stars 0 forks source link

Redis slave节点地址解析问题 #3

Open zshuangyan opened 6 years ago

zshuangyan commented 6 years ago

由于redis本身无法支持端口映射，直接部署redis sentinel到k8s集群上，Redis主节点的INFO信息中slave节点的ip和port分别是pod所在的node的ip和容器的port，如果要找到对应的从节点实例，应该是nodeip+nodeport或者而不是nodeip+containerport，并且Redis sentinel从主节点的INFO信息中获取从节点的ip和port，并更新到自身的配置文件中，因此sentinel获取的从节点实例的地址也是错误的。尝试通过配置从节点slave-announce-ip为域名，slave-announce-port为从节点service对应的nodeport暴露地址，镜像文件如下：
Dockerfile

FROM redis:3.2
MAINTAINER ZhangShuangyan <Zhangshuangyan@gridsum.com>

ADD docker-entrypoint.sh /
RUN chmod +x /docker-entrypoint.sh

ENV ANNOUNCE_IP redis-pro.internal.gridsumdissector.com
EXPOSE 6379

ENTRYPOINT ["/docker-entrypoint.sh"]

docker-entrypoint.sh

#!/bin/bash
set -e
CONFIGURATION_FILE=/etc/redis.conf

if [ "$ANNOUNCE_IP" ]; then
    echo "slave-announce-ip $ANNOUNCE_IP" >> $CONFIGURATION_FILE
fi

if [ "$ANNOUNCE_PORT" ]; then
    echo "slave-announce-port $ANNOUNCE_PORT" >> $CONFIGURATION_FILE
fi

if [ "$REDIS_MASTER" ]; then
    echo "slaveof $REDIS_MASTER 6379" >> $CONFIGURATION_FILE
fi

exec redis-server $CONFIGURATION_FILE

redis-slave-RC.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: "${redis_name}-slave"
  labels:
    name: "${redis_name}-slave"
spec:
  replicas: 1
  selector:
    name: "${redis_name}-slave"
  template:
    metadata:
      labels:
        name: "${redis_name}-slave"
        redis-rs-name: "${redis_name}-rainbow"
    spec:
      containers:
      - name: "${redis_name}-slave"
        image: repository.gridsum.com:8443/library/redis-announce:0.3
        env:
        - name: REDIS_MASTER
          value: "${redis_name}-master"
        - name: ANNOUNCE_PORT
          value: "${slave_nodeport}"
        ports:
        - containerPort: 6379
      nodeSelector:
        zone: "paas"

运行后sentinel日志如下：

1:X 08 Nov 08:03:46.635 # Sentinel ID is 5e628d429ba939e9281e6d3b177dd5769a30b768
1:X 08 Nov 08:03:46.635 # +monitor master mymaster 10.254.202.211 6379 quorum 2
1:X 08 Nov 08:03:51.693 # +sdown master mymaster 10.254.202.211 6379
1:X 08 Nov 08:04:01.757 # -sdown master mymaster 10.254.202.211 6379
1:X 08 Nov 08:04:01.909 * +sentinel sentinel f44079f807faa44d0acaa27b648cdee6ab0e22dd 17.0.111.7 26379 @ mymaster 10.254.202.211 6379
1:X 08 Nov 08:04:03.618 * +sentinel sentinel 77a4427aaca9054249382c56de88c5737ac3680d 17.0.111.6 26379 @ mymaster 10.254.202.211 6379
1:X 08 Nov 08:04:51.923 * +slave slave 17.0.112.4:6379 17.0.112.4 6379 @ mymaster 10.254.202.211 6379

可以看到sentinel看到的从节点的地址是17.0.112.4:6379，测试可以正常主备切换

但是本周测试发现，现在sentinel从master解析到的slave地址不再是17.0.112.4:6379这种形式的了，而是如下：

port 26379
sentinel myid 04cfb2bc1613a96aff5c2446ea01021b1d7aad59
sentinel monitor mymaster 10.254.163.89 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 120000
# Generated by CONFIG REWRITE
dir "/data"
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel known-slave mymaster 10.202.40.103 44983
sentinel known-sentinel mymaster 17.0.111.11 26379 1ea7a64248bffea81fef210ccfc4d3de14b21a21
sentinel known-sentinel mymaster 17.0.111.9 26379 542a2810fe992da5694bfa53663fe94fd4dd94fd
sentinel current-epoch 0

从sentinel的配置文件更新中，我们可以看到slave的地址被解析为10.202.40.103 44983，对master节点执行缩容到0的操作，然后观察结果如下：

zsy@ubuntu:~/app-market-public/kubernetes-templates/redis-replicaset/0$ redis-cli -h redis-pro.internal.gridsumdissector.com -p 50883 sentinel master mymaster
 1) "name"
 2) "mymaster"
 3) "ip"
 4) "10.202.40.102"
 5) "port"
 6) "44983"
 7) "runid"
 8) "7313e034fc98bcfbe5c2b60b8f030da376328961"
 9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "208"
19) "last-ping-reply"
20) "208"
21) "down-after-milliseconds"
22) "5000"
23) "info-refresh"
24) "3086"
25) "role-reported"
26) "slave"
27) "role-reported-time"
28) "13090"
29) "config-epoch"
30) "1"
31) "num-slaves"
32) "2"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "120000"
39) "parallel-syncs"
40) "1"
zsy@ubuntu:~/app-market-public/kubernetes-templates/redis-replicaset/0$ redis-cli -h redis-pro.internal.gridsumdissector.com -p 50883 sentinel slaves mymaster
1)  1) "name"
    2) "10.254.163.89:6379"
    3) "ip"
    4) "10.254.163.89"
    5) "port"
    6) "6379"
    7) "runid"
    8) ""
    9) "flags"
   10) "s_down,slave"
   11) "link-pending-commands"
   12) "100"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "39287"
   17) "last-ok-ping-reply"
   18) "39287"
   19) "last-ping-reply"
   20) "39287"
   21) "s-down-time"
   22) "34281"
   23) "down-after-milliseconds"
   24) "5000"
   25) "info-refresh"
   26) "1510548757609"
   27) "role-reported"
   28) "slave"
   29) "role-reported-time"
   30) "39287"
   31) "master-link-down-time"
   32) "0"
   33) "master-link-status"
   34) "err"
   35) "master-host"
   36) "?"
   37) "master-port"
   38) "0"
   39) "slave-priority"
   40) "100"
   41) "slave-repl-offset"
   42) "0"
2)  1) "name"
    2) "10.202.40.103:44983"
    3) "ip"
    4) "10.202.40.103"
    5) "port"
    6) "44983"
    7) "runid"
    8) "7313e034fc98bcfbe5c2b60b8f030da376328961"
    9) "flags"
   10) "slave"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "518"
   19) "last-ping-reply"
   20) "518"
   21) "down-after-milliseconds"
   22) "5000"
   23) "info-refresh"
   24) "417"
   25) "role-reported"
   26) "slave"
   27) "role-reported-time"
   28) "19206"
   29) "master-link-down-time"
   30) "1510548757000"
   31) "master-link-status"
   32) "err"
   33) "master-host"
   34) "10.202.40.102"
   35) "master-port"
   36) "44983"
   37) "slave-priority"
   38) "100"
   39) "slave-repl-offset"
   40) "1"

这里出现一个新选出的主节点10.202.40.102：44983，还出现一个从节点10.202.40.103:44983，而其实我们通过执行nslookup命令可以知道域名redis-pro.internal.gridsumdissector.com就是对应着这两个地址：10.202.40.102和10.202.40.103

nslookup redis-pro.internal.gridsumdissector.com
Server:     127.0.1.1
Address:    127.0.1.1#53

Non-authoritative answer:
Name:   redis-pro.internal.gridsumdissector.com
Address: 10.202.40.102
Name:   redis-pro.internal.gridsumdissector.com
Address: 10.202.40.103

但是我们的sentinel以为它们是两个实例，所以sentinel对10.202.40.102：44983执行了slaveof 10.202.40.103 44983的命令，接下来我们看到结果如下：

1:S 13 Nov 05:20:48.274 * Connecting to MASTER 10.202.40.102:44983
1:S 13 Nov 05:20:48.274 * MASTER <-> SLAVE sync started
1:S 13 Nov 05:20:48.278 * Non blocking connect for SYNC fired the event.
1:S 13 Nov 05:20:48.278 * Master replied to PING, replication can continue...
1:S 13 Nov 05:20:48.280 * Partial resynchronization not possible (no cached master)
1:S 13 Nov 05:20:48.280 * Master does not support PSYNC or is in error state (reply: -ERR Can't SYNC while not connected with my master)
1:S 13 Nov 05:20:48.280 * Retrying with SYNC...
1:S 13 Nov 05:20:48.281 # MASTER aborted replication with an error: ERR Can't SYNC while not connected with my master
1:S 13 Nov 05:20:49.276 * Connecting to MASTER 10.202.40.102:44983
1:S 13 Nov 05:20:49.276 * MASTER <-> SLAVE sync started
1:S 13 Nov 05:20:49.277 * Non blocking connect for SYNC fired the event.
1:S 13 Nov 05:20:49.278 * Master replied to PING, replication can continue...
1:S 13 Nov 05:20:49.282 * Partial resynchronization not possible (no cached master)
1:S 13 Nov 05:20:49.283 * Master does not support PSYNC or is in error state (reply: -ERR Can't SYNC while not connected with my master)
1:S 13 Nov 05:20:49.283 * Retrying with SYNC...
1:S 13 Nov 05:20:49.284 # MASTER aborted replication with an error: ERR Can't SYNC while not connected with my master
1:S 13 Nov 05:20:50.279 * Connecting to MASTER 10.202.40.102:44983
1:S 13 Nov 05:20:50.279 * MASTER <-> SLAVE sync started
1:S 13 Nov 05:20:50.280 * Non blocking connect for SYNC fired the event.
1:S 13 Nov 05:20:50.281 * Master replied to PING, replication can continue...
1:S 13 Nov 05:20:50.282 * Partial resynchronization not possible (no cached master)
1:S 13 Nov 05:20:50.283 * Master does not support PSYNC or is in error state (reply: -ERR Can't SYNC while not connected with my master)
1:S 13 Nov 05:20:50.283 * Retrying with SYNC...
1:S 13 Nov 05:20:50.283 # MASTER aborted replication with an error: ERR Can't SYNC while not connected with my master

这个实例进入了slave的状态，Redis集群中没有了主节点

zsy@ubuntu:~/app-market-public/kubernetes-templates/redis-replicaset/0$ redis-cli -h redis-pro.internal.gridsumdissector.com -p 50883 sentinel master mymaster
 1) "name"
 2) "mymaster"
 3) "ip"
 4) "10.202.40.102"
 5) "port"
 6) "44983"
 7) "runid"
 8) "7313e034fc98bcfbe5c2b60b8f030da376328961"
 9) "flags"
10) "s_down,o_down,master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "255"
19) "last-ping-reply"
20) "255"
21) "s-down-time"
22) "86900"
23) "o-down-time"
24) "86838"
25) "down-after-milliseconds"
26) "5000"
27) "info-refresh"
28) "1495"
29) "role-reported"
30) "slave"
31) "role-reported-time"
32) "111906"
33) "config-epoch"
34) "1"
35) "num-slaves"
36) "2"
37) "num-other-sentinels"
38) "2"
39) "quorum"
40) "2"
41) "failover-timeout"
42) "120000"
43) "parallel-syncs"
44) "1"

解决的办法是从域名对应的ip地址中选择其中一个，比如10.202.40.102作为slave-announce-ip