failover

[promzeus]

I recommend adding the following logic to your code: In case of changing the ip master, we change this ip in the endpoint. I made a working solution on sh, but still, if this functionality was in your operator, it would be much more convenient.

#!/bin/bash

function getMasterIP() {
    # echo "get sentinel master ip"
    MASTER_IP=$(echo "SENTINEL get-master-addr-by-name mymaster" |nc rfs-redis-node 26379 |grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' |sed "s/$(printf '\r')\$//")
}

function getKubeEndpointIP() {
    # echo "get kube endpoint ip"
    KUBE_IP=$(kubectl get ep/redis-headless -ojsonpath="{.subsets..addresses..ip}")
}

function setKubeEndpointIP() {
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Endpoints
metadata:
  name: redis-headless
  labels:
    k8s-app: redis-headless
subsets:
- addresses:
  - ip: "$MASTER_IP"
  ports:
  - name: redis-headless
    port: 6379
    protocol: TCP
EOF
}

while :
do

DATE=$(date +"%Y-%m-%d %H:%M:%S")

getMasterIP
getKubeEndpointIP

if [ -z $MASTER_IP ]; then
        echo "$DATE sentinel master ip is empty, sleep 10s"
        echo MasterIP: $MASTER_IP
    else
        # echo "$DATE sentinel master ip is not empty, run check kube endpoint ip"
    if [ -z $KUBE_IP ]; then
            echo "$DATE kube endpoint ip is empty, sleep 10s"
            echo KubeEndpointIP: $KUBE_IP
        else
            # echo "$DATE kube endpoint ip is not empty, run check master ip == kube ip"
            if [ "$MASTER_IP" == "$KUBE_IP" ]; then
                echo "$DATE  Nothing to change. Sentinel master ip:$MASTER_IP is equal to kube endpoint ip:$KUBE_IP, sleep 10s"
            else
                echo "$DATE sentinel master ip is not equal to kube endpoint ip"
                echo "$DATE Run set $MASTER_IP to kube epoint"
                setKubeEndpointIP
                echo MasterIP: ${MASTER_IP}
                echo KubeEndpointIP: ${KUBE_IP}
        fi
    fi
fi

sleep 10

done

The very idea and logic of work you can see in my helm chart. https://github.com/promzeus/RedisOperator-haproxy-failover The chart also has a haproxy failover that has worked for us for almost a year. But the loads are growing and haproxy is no longer enough, there is no point in discussing its bugs and problems with closing connections, caching, etc.

[samof76]

https://github.com/spotahome/redis-operator/pull/419 This should ideally fix it, if you have a service looking up the right label. But you would need to take this with a pinch of salt, and these labels are reconcile on a delta time from the actual failover. If you would want some consistent mechanism, you would to subscribe to sentinel failover events, and make appropriate decisions.

You could also look at https://github.com/redis/redis/blob/bed6d759bcb1ddcdab76da1b46a120e2e610036b/sentinel.conf#L270-L278

[github-actions[bot]]

This issue is stale because it has been open for 45 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.