spotahome / redis-operator

Redis Operator creates/configures/manages high availability redis with sentinel automatic failover atop Kubernetes.
Apache License 2.0
1.5k stars 356 forks source link

Issue mismatch number of sentinels in memory #545

Closed Braineanear closed 1 year ago

Braineanear commented 1 year ago

Actual behaviour

Getting warning messages on redis operator logs after upgrading to 1.2.4

Sentinel "ip" mismatch number of sentinels in memory. resetting
Sentinel "ip" mismatch number of expected slaves in memory. resetting
Fixing sentinel not monitoring expected master: sentinel monitoring 127.0.0.1:6379 instead ip:6379

Steps to reproduce the behaviour

Upgraded to version 1.2.4

Environment

How are the pieces configured?

Logs


time="2022-12-29T13:21:55Z" level=warning msg="Sentinel "ip" mismatch number of expected slaves in memory. resetting" namespace=redis redisfailover=redis src="checker.go:230"
time="2022-12-29T13:21:55Z" level=warning msg="Sentinel "ip" mismatch number of expected slaves in memory. resetting" namespace=redis redisfailover=redis src="checker.go:230"
time="2022-12-29T13:21:55Z" level=warning msg="Sentinel "ip" mismatch number of expected slaves in memory. resetting" namespace=redis redisfailover=redis src="checker.go:230"
time="2022-12-29T13:23:24Z" level=warning msg="Number of Masters running is 0" namespace=redis redisfailover=redis src="handler.go:79"
time="2022-12-29T13:23:24Z" level=info msg="No master avaiable but max pod up time is : 40.000000" namespace=redis redisfailover=redis src="handler.go:79"
time="2022-12-29T13:23:24Z" level=warning msg="Unable to get result for CKQUORUM comand" src="check.go:213"
time="2022-12-29T13:23:24Z" level=warning msg="Unable to get result for CKQUORUM comand" src="check.go:213"
time="2022-12-29T13:23:24Z" level=warning msg="Unable to get result for CKQUORUM comand" src="check.go:213"
time="2022-12-29T13:23:24Z" level=error msg="insufficnet sentinel to reach Quorum - Unhealthy count: 3" src="checker.go:146"
time="2022-12-29T13:23:24Z" level=warning msg="Quorum not available for sentinel to choose master,estimated unhealthy sentinels :3 , Operator to step-in" namespace=redis redisfailover=redis src="handler.go:79"
time="2022-12-29T13:23:24Z" level=info msg="New master is rfr-redis-1 with ip "ip"" namespace=redis redisfailover=redis service=redis.healer src="checker.go:150"
time="2022-12-29T13:23:24Z" level=info msg="Update pod label, namespace: redis, pod name: rfr-redis-1, labels: map[redisfailovers-role:master]" service=k8s.pod src="heal.go:52"
time="2022-12-29T13:23:24Z" level=info msg="Making pod rfr-redis-2 slave of "ip"" service=redis.healer src="checker.go:150"
time="2022-12-29T13:23:24Z" level=info msg="Making pod rfr-redis-0 slave of "ip"" service=redis.healer src="checker.go:150"
time="2022-12-29T13:23:25Z" level=warning msg="Fixing sentinel not monitoring expected master: sentinel monitoring 127.0.0.1:6379 instead "ip":6379" namespace=redis redisfailover=redis src="handler.go:79"
time="2022-12-29T13:23:25Z" level=warning msg="Fixing sentinel not monitoring expected master: sentinel monitoring 127.0.0.1:6379 instead "ip":6379" namespace=redis redisfailover=redis src="handler.go:79"
time="2022-12-29T13:23:25Z" level=warning msg="Fixing sentinel not monitoring expected master: sentinel monitoring 127.0.0.1:6379 instead "ip":6379" namespace=redis redisfailover=redis src="handler.go:79"
time="2022-12-29T13:23:25Z" level=warning msg="Sentinel "ip" mismatch number of sentinels in memory. resetting" namespace=redis redisfailover=redis src="checker.go:230"
time="2022-12-29T13:23:25Z" level=warning msg="Sentinel "ip" mismatch number of sentinels in memory. resetting" namespace=redis redisfailover=redis src="checker.go:230"
time="2022-12-29T13:23:25Z" level=warning msg="Sentinel "ip" mismatch number of sentinels in memory. resetting" namespace=redis redisfailover=redis src="checker.go:230"
time="2022-12-29T13:23:25Z" level=warning msg="Sentinel "ip" mismatch number of expected slaves in memory. resetting" namespace=redis redisfailover=redis src="checker.go:230"
time="2022-12-29T13:23:25Z" level=warning msg="Sentinel "ip" mismatch number of expected slaves in memory. resetting" namespace=redis redisfailover=redis src="checker.go:230"
time="2022-12-29T13:23:25Z" level=warning msg="Sentinel "ip" mismatch number of expected slaves in memory. resetting" namespace=redis redisfailover=redis src="checker.go:230"
github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 45 days with no activity.

github-actions[bot] commented 1 year ago

This issue was closed because it has been inactive for 14 days since being marked as stale.