Bitnami/Redis resurrect master node after failover and reconnect with new master

[ronaldvanderrest]

Which chart: Redis

Describe the bug: After a slave has become the new master due to a failover, a resurrected master pod does not become slave to this master.

Describe the bug: In a setup with one master-node, 3 slave nodes and corresponding sentinels, the following occurs:

1. first there is a healthy situation, where the master has a master role, and has the three slaves connected.
2. when the master fails, the sentinels elect any of the slaves to become the new master
3. the master node restarts, but does not communicate with the slaves. So it starts an entirely new cluster. Result is that there are now two running clusters.

So currently there I'm unaware of a graceful way to resurrect the master statefulset and make it part of the cluster again. It seems only slaves are allowed to ressurect.

The deployment should be able to withstand restarts, for example when the underlying NodeGroup is upgraded.

In an older commit of the redis-ha project, it was actually mentioned that Redis master only has bootstrapping purposes.

https://github.com/helm/charts/pull/7323/files/4984e0d8da5c56e9ffe97aba54e92ce02936fe47 " This package provides a highly available Redis cluster with multiple sentinels and standbys. Note the redis-master pod is used for bootstrapping only and can be deleted once the cluster is up and running. "

[andresbono]

Thanks for opening this issue @ronaldvanderrest. We would like to reproduce the issue on our side. Can you tell us the specific chart parameters that you set when you launched it?

[sachinsaju]

@ronaldvanderrest May be you can scale down the master stateful set after the chart installation and delete it.

[ronaldvanderrest]

@andresbono The used values file looks like this:

values file

```yaml ## Global Docker image parameters ## Please, note that this will override the image parameters, including dependencies, configured to use the global value ## Current available global Docker image parameters: imageRegistry and imagePullSecrets ## global: # imageRegistry: myRegistryName # imagePullSecrets: # - myRegistryKeySecretName # storageClass: myStorageClass redis: {} ## Bitnami Redis image version ## ref: https://hub.docker.com/r/bitnami/redis/tags/ ## image: registry: cnc-fdn-docker.artifactory.prd.ss.aws.insim.biz repository: redis ## Bitnami Redis image tag ## ref: https://github.com/bitnami/bitnami-docker-redis#supported-tags-and-respective-dockerfile-links ## tag: 6.0.1-debian-10-r1 ## Specify a imagePullPolicy ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images ## pullPolicy: IfNotPresent ## Optionally specify an array of imagePullSecrets. ## Secrets must be manually created in the namespace. ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ ## # pullSecrets: # - myRegistryKeySecretName ## String to partially override redis.fullname template (will maintain the release name) ## # nameOverride: ## String to fully override redis.fullname template ## # fullnameOverride: ## Cluster settings cluster: enabled: true slaveCount: 3 ## Use redis sentinel in the redis pod. This will disable the master and slave services and ## create one redis service with ports to the sentinel and the redis instances sentinel: enabled: true ## Require password authentication on the sentinel itself ## ref: https://redis.io/topics/sentinel usePassword: true ## Bitnami Redis Sentintel image version ## ref: https://hub.docker.com/r/bitnami/redis-sentinel/tags/ ## image: registry: cnc-fdn-docker.artifactory.prd.ss.aws.insim.biz repository: redis-sentinel ## Bitnami Redis image tag ## ref: https://github.com/bitnami/bitnami-docker-redis-sentinel#supported-tags-and-respective-dockerfile-links ## tag: 6.0.1-debian-10-r0 ## Specify a imagePullPolicy ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images ## pullPolicy: IfNotPresent ## Optionally specify an array of imagePullSecrets. ## Secrets must be manually created in the namespace. ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ ## # pullSecrets: # - myRegistryKeySecretName masterSet: mymaster initialCheckTimeout: 5 quorum: 2 downAfterMilliseconds: 60000 failoverTimeout: 18000 parallelSyncs: 1 port: 26379 ## Additional Redis configuration for the sentinel nodes ## ref: https://redis.io/topics/config ## configmap: ## Enable or disable static sentinel IDs for each replicas ## If disabled each sentinel will generate a random id at startup ## If enabled, each replicas will have a constant ID on each start-up ## staticID: false ## Configure extra options for Redis Sentinel liveness and readiness probes ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes) ## livenessProbe: enabled: true initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: enabled: true initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 5 customLivenessProbe: {} customReadinessProbe: {} ## Redis Sentinel resource requests and limits ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ # resources: # requests: # memory: 256Mi # cpu: 100m ## Redis Sentinel Service properties service: ## Redis Sentinel Service type type: ClusterIP sentinelPort: 26379 redisPort: 6379 ## Specify the nodePort value for the LoadBalancer and NodePort service types. ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport ## # sentinelNodePort: # redisNodePort: ## Provide any additional annotations which may be required. This can be used to ## set the LoadBalancer service type to internal only. ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer ## annotations: {} labels: {} loadBalancerIP: ## Specifies the Kubernetes Cluster's Domain Name. ## clusterDomain: cluster.local networkPolicy: ## Specifies whether a NetworkPolicy should be created ## enabled: true ## The Policy model to apply. When set to false, only pods with the correct ## client label will have network access to the port Redis is listening ## on. When true, Redis will accept connections from any source ## (with the correct destination port). ## # allowExternal: true ## Allow connections from other namespacess. Just set label for namespace and set label for pods (optional). ## ingressNSMatchLabels: {} ingressNSPodMatchLabels: {} serviceAccount: ## Specifies whether a ServiceAccount should be created ## create: false ## The name of the ServiceAccount to use. ## If not set and create is true, a name is generated using the fullname template name: rbac: ## Specifies whether RBAC resources should be created ## create: false role: ## Rules to create. It follows the role specification # rules: # - apiGroups: # - extensions # resources: # - podsecuritypolicies # verbs: # - use # resourceNames: # - gce.unprivileged rules: [] ## Redis pod Security Context securityContext: enabled: true fsGroup: 1001 runAsUser: 1001 ## sysctl settings for master and slave pods ## ## Uncomment the setting below to increase the net.core.somaxconn value ## # sysctls: # - name: net.core.somaxconn # value: "10000" ## Use password authentication usePassword: true ## Redis password (both master and slave) ## Defaults to a random 10-character alphanumeric string if not set and usePassword is true ## ref: https://github.com/bitnami/bitnami-docker-redis#setting-the-server-password-on-first-run ## password: ## Use existing secret (ignores previous password) # existingSecret: ## Password key to be retrieved from Redis secret ## # existingSecretPasswordKey: ## Mount secrets as files instead of environment variables usePasswordFile: false ## Persist data to a persistent volume (Redis Master) persistence: ## A manually managed Persistent Volume and Claim ## Requires persistence.enabled: true ## If defined, PVC must be created manually before volume will be bound existingClaim: # Redis port redisPort: 6379 ## ## Redis Master parameters ## master: ## Redis command arguments ## ## Can be used to specify command line arguments, for example: ## command: "/run.sh" ## Additional Redis configuration for the master nodes ## ref: https://redis.io/topics/config ## configmap: ## Redis additional command line flags ## ## Can be used to specify command line flags, for example: ## ## extraFlags: ## - "--maxmemory-policy volatile-ttl" ## - "--repl-backlog-size 1024mb" extraFlags: [] ## Comma-separated list of Redis commands to disable ## ## Can be used to disable Redis commands for security reasons. ## Commands will be completely disabled by renaming each to an empty string. ## ref: https://redis.io/topics/security#disabling-of-specific-commands ## disableCommands: - FLUSHDB - FLUSHALL ## Redis Master additional pod labels and annotations ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ podLabels: {} podAnnotations: {} ## Redis Master resource requests and limits ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ # resources: # requests: # memory: 256Mi # cpu: 100m ## Use an alternate scheduler, e.g. "stork". ## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ ## # schedulerName: ## Configure extra options for Redis Master liveness and readiness probes ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes) ## livenessProbe: enabled: true initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: enabled: true initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 5 ## Configure custom probes for images other images like ## rhscl/redis-32-rhel7 rhscl/redis-5-rhel7 ## Only used if readinessProbe.enabled: false / livenessProbe.enabled: false ## # customLivenessProbe: # tcpSocket: # port: 6379 # initialDelaySeconds: 10 # periodSeconds: 5 # customReadinessProbe: # initialDelaySeconds: 30 # periodSeconds: 10 # timeoutSeconds: 5 # exec: # command: # - "container-entrypoint" # - "bash" # - "-c" # - "redis-cli set liveness-probe \"`date`\" | grep OK" customLivenessProbe: {} customReadinessProbe: {} ## Redis Master Node selectors and tolerations for pod assignment ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature ## # nodeSelector: {"beta.kubernetes.io/arch": "amd64"} # tolerations: [] ## Redis Master pod/node affinity/anti-affinity ## affinity: {} ## Redis Master Service properties service: ## Redis Master Service type type: ClusterIP port: 6379 ## Specify the nodePort value for the LoadBalancer and NodePort service types. ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport ## # nodePort: ## Provide any additional annotations which may be required. This can be used to ## set the LoadBalancer service type to internal only. ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer ## annotations: {} labels: {} loadBalancerIP: # loadBalancerSourceRanges: ["10.0.0.0/8"] ## Enable persistence using Persistent Volume Claims ## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/ ## persistence: enabled: true ## The path the volume will be mounted at, useful when using different ## Redis images. path: /data ## The subdirectory of the volume to mount to, useful in dev environments ## and one PV for multiple services. subPath: "" ## redis data Persistent Volume Storage Class ## If defined, storageClassName: ## If set to "-", storageClassName: "", which disables dynamic provisioning ## If undefined (the default) or set to null, no storageClassName spec is ## set, choosing the default provisioner. (gp2 on AWS, standard on ## GKE, AWS & OpenStack) ## # storageClass: "-" accessModes: - ReadWriteOnce size: 8Gi ## Persistent Volume selectors ## https://kubernetes.io/docs/concepts/storage/persistent-volumes/#selector matchLabels: {} matchExpressions: {} ## Update strategy, can be set to RollingUpdate or onDelete by default. ## https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets statefulset: updateStrategy: RollingUpdate ## Partition update strategy ## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions # rollingUpdatePartition: ## Redis Master pod priorityClassName ## priorityClassName: {} ## ## Redis Slave properties ## Note: service.type is a mandatory parameter ## The rest of the parameters are either optional or, if undefined, will inherit those declared in Redis Master ## slave: ## Slave Service properties service: ## Redis Slave Service type type: ClusterIP ## Redis port port: 6379 ## Specify the nodePort value for the LoadBalancer and NodePort service types. ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport ## # nodePort: ## Provide any additional annotations which may be required. This can be used to ## set the LoadBalancer service type to internal only. ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer ## annotations: {} labels: {} loadBalancerIP: # loadBalancerSourceRanges: ["10.0.0.0/8"] ## Redis slave port port: 6379 ## Can be used to specify command line arguments, for example: ## command: "/run.sh" ## Additional Redis configuration for the slave nodes ## ref: https://redis.io/topics/config ## configmap: ## Redis extra flags extraFlags: [] ## List of Redis commands to disable disableCommands: - FLUSHDB - FLUSHALL ## Redis Slave pod/node affinity/anti-affinity ## affinity: {} ## Configure extra options for Redis Slave liveness and readiness probes ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes) ## livenessProbe: enabled: true initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: enabled: true initialDelaySeconds: 5 periodSeconds: 10 timeoutSeconds: 10 successThreshold: 1 failureThreshold: 5 ## Configure custom probes for images other images like ## rhscl/redis-32-rhel7 rhscl/redis-5-rhel7 ## Only used if readinessProbe.enabled: false / livenessProbe.enabled: false ## # customLivenessProbe: # tcpSocket: # port: 6379 # initialDelaySeconds: 10 # periodSeconds: 5 # customReadinessProbe: # initialDelaySeconds: 30 # periodSeconds: 10 # timeoutSeconds: 5 # exec: # command: # - "container-entrypoint" # - "bash" # - "-c" # - "redis-cli set liveness-probe \"`date`\" | grep OK" customLivenessProbe: {} customReadinessProbe: {} ## Redis slave Resource # resources: # requests: # memory: 256Mi # cpu: 100m ## Redis slave selectors and tolerations for pod assignment # nodeSelector: {"beta.kubernetes.io/arch": "amd64"} # tolerations: [] ## Use an alternate scheduler, e.g. "stork". ## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ ## # schedulerName: ## Redis slave pod Annotation and Labels podLabels: {} podAnnotations: {} ## Redis slave pod priorityClassName # priorityClassName: {} ## Enable persistence using Persistent Volume Claims ## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/ ## persistence: enabled: true ## The path the volume will be mounted at, useful when using different ## Redis images. path: /data ## The subdirectory of the volume to mount to, useful in dev environments ## and one PV for multiple services. subPath: "" ## redis data Persistent Volume Storage Class ## If defined, storageClassName: ## If set to "-", storageClassName: "", which disables dynamic provisioning ## If undefined (the default) or set to null, no storageClassName spec is ## set, choosing the default provisioner. (gp2 on AWS, standard on ## GKE, AWS & OpenStack) ## # storageClass: "-" accessModes: - ReadWriteOnce size: 8Gi ## Persistent Volume selectors ## https://kubernetes.io/docs/concepts/storage/persistent-volumes/#selector matchLabels: {} matchExpressions: {} ## Update strategy, can be set to RollingUpdate or onDelete by default. ## https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets statefulset: updateStrategy: RollingUpdate ## Partition update strategy ## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions # rollingUpdatePartition: ## Prometheus Exporter / Metrics ## metrics: enabled: true image: registry: cnc-fdn-docker.artifactory.prd.ss.aws.insim.biz repository: redis-exporter tag: 1.6.0-debian-10-r2 pullPolicy: IfNotPresent ## Optionally specify an array of imagePullSecrets. ## Secrets must be manually created in the namespace. ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ ## # pullSecrets: # - myRegistryKeySecretName ## Metrics exporter resource requests and limits ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ ## # resources: {} ## Extra arguments for Metrics exporter, for example: ## extraArgs: ## check-keys: myKey,myOtherKey # extraArgs: {} ## Metrics exporter pod Annotation and Labels podAnnotations: prometheus.io/scrape: "true" prometheus.io/port: "9121" # podLabels: {} # Enable this if you're using https://github.com/coreos/prometheus-operator serviceMonitor: enabled: false ## Specify a namespace if needed # namespace: monitoring # fallback to the prometheus default unless specified # interval: 10s ## Defaults to what's used if you follow CoreOS [Prometheus Install Instructions](https://github.com/bitnami/charts/tree/master/bitnami/prometheus-operator#tldr) ## [Prometheus Selector Label](https://github.com/bitnami/charts/tree/master/bitnami/prometheus-operator#prometheus-operator-1) ## [Kube Prometheus Selector Label](https://github.com/bitnami/charts/tree/master/bitnami/prometheus-operator#exporters) selector: prometheus: kube-prometheus ## Custom PrometheusRule to be defined ## The value is evaluated as a template, so, for example, the value can depend on .Release or .Chart ## ref: https://github.com/coreos/prometheus-operator#customresourcedefinitions prometheusRule: enabled: false additionalLabels: {} namespace: "" ## Redis prometheus rules ## These are just examples rules, please adapt them to your needs. ## Make sure to constraint the rules to the current postgresql service. # rules: # - alert: RedisDown # expr: redis_up{service="{{ template "redis.fullname" . }}-metrics"} == 0 # for: 2m # labels: # severity: error # annotations: # summary: Redis instance {{ "{{ $labels.instance }}" }} down # description: Redis instance {{ "{{ $labels.instance }}" }} is down # - alert: RedisMemoryHigh # expr: > # redis_memory_used_bytes{service="{{ template "redis.fullname" . }}-metrics"} * 100 # / # redis_memory_max_bytes{service="{{ template "redis.fullname" . }}-metrics"} # > 90 =< 100 # for: 2m # labels: # severity: error # annotations: # summary: Redis instance {{ "{{ $labels.instance }}" }} is using too much memory # description: | # Redis instance {{ "{{ $labels.instance }}" }} is using {{ "{{ $value }}" }}% of its available memory. # - alert: RedisKeyEviction # expr: | # increase(redis_evicted_keys_total{service="{{ template "redis.fullname" . }}-metrics"}[5m]) > 0 # for: 1s # labels: # severity: error # annotations: # summary: Redis instance {{ "{{ $labels.instance }}" }} has evicted keys # description: | # Redis instance {{ "{{ $labels.instance }}" }} has evicted {{ "{{ $value }}" }} keys in the last 5 minutes. rules: [] ## Metrics exporter pod priorityClassName # priorityClassName: {} service: type: ClusterIP ## Use serviceLoadBalancerIP to request a specific static IP, ## otherwise leave blank # loadBalancerIP: annotations: {} labels: {} ## ## Init containers parameters: ## volumePermissions: Change the owner of the persist volume mountpoint to RunAsUser:fsGroup ## volumePermissions: enabled: false image: registry: docker.io repository: bitnami/minideb tag: buster pullPolicy: Always ## Optionally specify an array of imagePullSecrets. ## Secrets must be manually created in the namespace. ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ ## # pullSecrets: # - myRegistryKeySecretName resources: {} # resources: # requests: # memory: 128Mi # cpu: 100m ## Redis config file ## ref: https://redis.io/topics/config ## configmap: |- # Enable AOF https://redis.io/topics/persistence#append-only-file appendonly yes # Disable RDB persistence, AOF persistence already enabled. save "" ## Sysctl InitContainer ## used to perform sysctl operation to modify Kernel settings (needed sometimes to avoid warnings) sysctlImage: enabled: false command: [] registry: docker.io repository: bitnami/minideb tag: buster pullPolicy: Always ## Optionally specify an array of imagePullSecrets. ## Secrets must be manually created in the namespace. ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ ## # pullSecrets: # - myRegistryKeySecretName mountHostSys: false resources: {} # resources: # requests: # memory: 128Mi # cpu: 100m ## PodSecurityPolicy configuration ## ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/ ## podSecurityPolicy: ## Specifies whether a PodSecurityPolicy should be created ## create: false ```

The entire deployed helm manifest results in:

```yaml # Source: redis/templates/networkpolicy.yaml kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: redis-fixredis3 namespace: core-e labels: app: redis chart: redis-10.6.13 release: redis-fixredis3 heritage: Helm spec: podSelector: matchLabels: app: redis release: redis-fixredis3 policyTypes: - Ingress - Egress egress: # Allow dns resolution - ports: - port: 53 protocol: UDP # Allow outbound connections to other cluster pods - ports: - port: 6379 - port: 26379 to: - podSelector: matchLabels: app: redis release: redis-fixredis3 ingress: # Allow inbound connections - ports: - port: 6379 - port: 26379 from: - podSelector: matchLabels: redis-fixredis3-client: "true" - podSelector: matchLabels: app: redis release: redis-fixredis3 # Allow prometheus scrapes for metrics - ports: - port: 9121 --- # Source: redis/templates/secret.yaml apiVersion: v1 kind: Secret metadata: name: redis-fixredis3 namespace: core-e labels: app: redis chart: redis-10.6.13 release: "redis-fixredis3" heritage: "Helm" type: Opaque data: redis-password: "b1c5RENEYlp6VQ==" --- # Source: redis/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: redis-fixredis3 namespace: core-e labels: app: redis chart: redis-10.6.13 heritage: Helm release: redis-fixredis3 data: redis.conf: |- # User-supplied configuration: # Enable AOF https://redis.io/topics/persistence#append-only-file appendonly yes # Disable RDB persistence, AOF persistence already enabled. save "" master.conf: |- dir /data rename-command FLUSHDB "" rename-command FLUSHALL "" replica.conf: |- dir /data slave-read-only yes rename-command FLUSHDB "" rename-command FLUSHALL "" sentinel.conf: |- dir "/tmp" bind 0.0.0.0 port 26379 sentinel monitor mymaster redis-fixredis3-master-0.redis-fixredis3-headless.core-e.svc.cluster.local 6379 2 sentinel down-after-milliseconds mymaster 60000 sentinel failover-timeout mymaster 18000 sentinel parallel-syncs mymaster 1 --- # Source: redis/templates/health-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: redis-fixredis3-health namespace: core-e labels: app: redis chart: redis-10.6.13 heritage: Helm release: redis-fixredis3 data: ping_readiness_local.sh: |- #!/bin/bash no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.*) ]] && echo --no-auth-warning) response=$( timeout -s 3 $1 \ redis-cli \ -a $REDIS_PASSWORD $no_auth_warning \ -h localhost \ -p $REDIS_PORT \ ping ) if [ "$response" != "PONG" ]; then echo "$response" exit 1 fi ping_liveness_local.sh: |- #!/bin/bash no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.*) ]] && echo --no-auth-warning) response=$( timeout -s 3 $1 \ redis-cli \ -a $REDIS_PASSWORD $no_auth_warning \ -h localhost \ -p $REDIS_PORT \ ping ) if [ "$response" != "PONG" ] && [ "$response" != "LOADING Redis is loading the dataset in memory" ]; then echo "$response" exit 1 fi ping_sentinel.sh: |- #!/bin/bash no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.*) ]] && echo --no-auth-warning) response=$( timeout -s 3 $1 \ redis-cli \ -a $REDIS_PASSWORD $no_auth_warning \ -h localhost \ -p $REDIS_SENTINEL_PORT \ ping ) if [ "$response" != "PONG" ]; then echo "$response" exit 1 fi parse_sentinels.awk: |- /ip/ {FOUND_IP=1} /port/ {FOUND_PORT=1} /runid/ {FOUND_RUNID=1} !/ip|port|runid/ { if (FOUND_IP==1) { IP=$1; FOUND_IP=0; } else if (FOUND_PORT==1) { PORT=$1; FOUND_PORT=0; } else if (FOUND_RUNID==1) { printf "\nsentinel known-sentinel mymaster %s %s %s", IP, PORT, $0; FOUND_RUNID=0; } } ping_readiness_master.sh: |- #!/bin/bash no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.*) ]] && echo --no-auth-warning) response=$( timeout -s 3 $1 \ redis-cli \ -a $REDIS_MASTER_PASSWORD $no_auth_warning \ -h $REDIS_MASTER_HOST \ -p $REDIS_MASTER_PORT_NUMBER \ ping ) if [ "$response" != "PONG" ]; then echo "$response" exit 1 fi ping_liveness_master.sh: |- #!/bin/bash no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.*) ]] && echo --no-auth-warning) response=$( timeout -s 3 $1 \ redis-cli \ -a $REDIS_MASTER_PASSWORD $no_auth_warning \ -h $REDIS_MASTER_HOST \ -p $REDIS_MASTER_PORT_NUMBER \ ping ) if [ "$response" != "PONG" ] && [ "$response" != "LOADING Redis is loading the dataset in memory" ]; then echo "$response" exit 1 fi ping_readiness_local_and_master.sh: |- script_dir="$(dirname "$0")" exit_status=0 "$script_dir/ping_readiness_local.sh" $1 || exit_status=$? "$script_dir/ping_readiness_master.sh" $1 || exit_status=$? exit $exit_status ping_liveness_local_and_master.sh: |- script_dir="$(dirname "$0")" exit_status=0 "$script_dir/ping_liveness_local.sh" $1 || exit_status=$? "$script_dir/ping_liveness_master.sh" $1 || exit_status=$? exit $exit_status --- # Source: redis/templates/headless-svc.yaml apiVersion: v1 kind: Service metadata: name: redis-fixredis3-headless namespace: core-e labels: app: redis chart: redis-10.6.13 release: redis-fixredis3 heritage: Helm spec: type: ClusterIP clusterIP: None ports: - name: redis port: 6379 targetPort: redis - name: redis-sentinel port: 26379 targetPort: redis-sentinel selector: app: redis release: redis-fixredis3 --- # Source: redis/templates/metrics-svc.yaml apiVersion: v1 kind: Service metadata: name: redis-fixredis3-metrics namespace: core-e labels: app: redis chart: redis-10.6.13 release: redis-fixredis3 heritage: Helm app.kubernetes.io/component: "metrics" spec: type: ClusterIP ports: - name: metrics port: 9121 targetPort: metrics selector: app: redis release: redis-fixredis3 --- # Source: redis/templates/redis-with-sentinel-svc.yaml apiVersion: v1 kind: Service metadata: name: redis-fixredis3 namespace: core-e labels: app: redis chart: redis-10.6.13 release: redis-fixredis3 heritage: Helm spec: type: ClusterIP ports: - name: redis port: 6379 targetPort: redis - name: redis-sentinel port: 26379 targetPort: redis-sentinel selector: app: redis release: redis-fixredis3 --- # Source: redis/templates/redis-master-statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: redis-fixredis3-master namespace: core-e labels: app: redis chart: redis-10.6.13 release: redis-fixredis3 heritage: Helm spec: selector: matchLabels: app: redis release: redis-fixredis3 role: master serviceName: redis-fixredis3-headless template: metadata: labels: app: redis chart: redis-10.6.13 release: redis-fixredis3 role: master annotations: checksum/health: 6bb3f32ea26d3964f4b160257cc96cc031c1b08f336a4311890262f5cc0de5bc checksum/configmap: 4dd51b5b0edf6772cc5f710797b1b59226f11e31198105178d7e0f9220bb1917 checksum/secret: 8788f13092ff659875e1bb4224227bd3e0144d3c08181f63958b82d1292d9761 prometheus.io/port: "9121" prometheus.io/scrape: "true" spec: securityContext: fsGroup: 1001 serviceAccountName: default containers: - name: redis image: cnc-fdn-docker.artifactory.prd.ss.aws.insim.biz/redis:6.0.1-debian-10-r1 imagePullPolicy: "IfNotPresent" securityContext: runAsUser: 1001 command: - /bin/bash - -c - | if [[ -n $REDIS_PASSWORD_FILE ]]; then password_aux=`cat ${REDIS_PASSWORD_FILE}` export REDIS_PASSWORD=$password_aux fi if [[ ! -f /opt/bitnami/redis/etc/master.conf ]];then cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf fi if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf fi ARGS=("--port" "${REDIS_PORT}") ARGS+=("--requirepass" "${REDIS_PASSWORD}") ARGS+=("--masterauth" "${REDIS_PASSWORD}") ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf") ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf") /run.sh ${ARGS[@]} env: - name: REDIS_REPLICATION_MODE value: master - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-fixredis3 key: redis-password - name: REDIS_PORT value: "6379" ports: - name: redis containerPort: 6379 livenessProbe: initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 exec: command: - sh - -c - /health/ping_liveness_local.sh 5 readinessProbe: initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 5 exec: command: - sh - -c - /health/ping_readiness_local.sh 5 resources: null volumeMounts: - name: health mountPath: /health - name: redis-data mountPath: /data subPath: - name: config mountPath: /opt/bitnami/redis/mounted-etc - name: redis-tmp-conf mountPath: /opt/bitnami/redis/etc/ - name: sentinel image: "cnc-fdn-docker.artifactory.prd.ss.aws.insim.biz/redis-sentinel:6.0.1-debian-10-r0" imagePullPolicy: "IfNotPresent" securityContext: runAsUser: 1001 command: - /bin/bash - -c - | if [[ -n $REDIS_PASSWORD_FILE ]]; then password_aux=`cat ${REDIS_PASSWORD_FILE}` export REDIS_PASSWORD=$password_aux fi if [[ ! -f /opt/bitnami/redis-sentinel/etc/sentinel.conf ]];then cp /opt/bitnami/redis-sentinel/mounted-etc/sentinel.conf /opt/bitnami/redis-sentinel/etc/sentinel.conf printf "\nsentinel auth-pass mymaster $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf printf "\nrequirepass $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf fi echo "Getting information about current running sentinels" # Get information from existing sentinels existing_sentinels=$(timeout -s 3 5 redis-cli --raw -h redis-fixredis3 -a "$REDIS_PASSWORD" -p 26379 SENTINEL sentinels mymaster) echo "$existing_sentinels" | awk -f /health/parse_sentinels.awk | tee -a /opt/bitnami/redis-sentinel/etc/sentinel.conf redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-fixredis3 key: redis-password - name: REDIS_SENTINEL_PORT value: "26379" ports: - name: redis-sentinel containerPort: 26379 livenessProbe: initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 exec: command: - sh - -c - /health/ping_sentinel.sh 5 readinessProbe: initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 5 exec: command: - sh - -c - /health/ping_sentinel.sh 5 resources: null volumeMounts: - name: health mountPath: /health - name: redis-data mountPath: /data subPath: - name: config mountPath: /opt/bitnami/redis-sentinel/mounted-etc - name: sentinel-tmp-conf mountPath: /opt/bitnami/redis-sentinel/etc/ - name: metrics image: cnc-fdn-docker.artifactory.prd.ss.aws.insim.biz/redis-exporter:1.6.0-debian-10-r2 imagePullPolicy: "IfNotPresent" command: - /bin/bash - -c - | if [[ -f '/secrets/redis-password' ]]; then export REDIS_PASSWORD=$(cat /secrets/redis-password) fi redis_exporter env: - name: REDIS_ALIAS value: redis-fixredis3 - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-fixredis3 key: redis-password volumeMounts: ports: - name: metrics containerPort: 9121 resources: null volumes: - name: health configMap: name: redis-fixredis3-health defaultMode: 0755 - name: config configMap: name: redis-fixredis3 - name: redis-tmp-conf emptyDir: {} - name: sentinel-tmp-conf emptyDir: {} volumeClaimTemplates: - metadata: name: redis-data labels: app: redis release: redis-fixredis3 heritage: Helm component: master spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "8Gi" selector: updateStrategy: type: RollingUpdate --- # Source: redis/templates/redis-slave-statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: redis-fixredis3-slave namespace: core-e labels: app: redis chart: redis-10.6.13 release: redis-fixredis3 heritage: Helm spec: replicas: 3 serviceName: redis-fixredis3-headless selector: matchLabels: app: redis release: redis-fixredis3 role: slave template: metadata: labels: app: redis release: redis-fixredis3 chart: redis-10.6.13 role: slave annotations: checksum/health: 6bb3f32ea26d3964f4b160257cc96cc031c1b08f336a4311890262f5cc0de5bc checksum/configmap: 4dd51b5b0edf6772cc5f710797b1b59226f11e31198105178d7e0f9220bb1917 checksum/secret: 883aec12f192e3189e12beaa396b2464f1a1eb7065bbebeada0822ada0de9ee7 prometheus.io/port: "9121" prometheus.io/scrape: "true" spec: securityContext: fsGroup: 1001 serviceAccountName: default containers: - name: redis image: cnc-fdn-docker.artifactory.prd.ss.aws.insim.biz/redis:6.0.1-debian-10-r1 imagePullPolicy: "IfNotPresent" securityContext: runAsUser: 1001 command: - /bin/bash - -c - | if [[ -n $REDIS_PASSWORD_FILE ]]; then password_aux=`cat ${REDIS_PASSWORD_FILE}` export REDIS_PASSWORD=$password_aux fi if [[ -n $REDIS_MASTER_PASSWORD_FILE ]]; then password_aux=`cat ${REDIS_MASTER_PASSWORD_FILE}` export REDIS_MASTER_PASSWORD=$password_aux fi if [[ ! -f /opt/bitnami/redis/etc/replica.conf ]];then cp /opt/bitnami/redis/mounted-etc/replica.conf /opt/bitnami/redis/etc/replica.conf fi if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf fi ARGS=("--port" "${REDIS_PORT}") ARGS+=("--slaveof" "${REDIS_MASTER_HOST}" "${REDIS_MASTER_PORT_NUMBER}") ARGS+=("--requirepass" "${REDIS_PASSWORD}") ARGS+=("--masterauth" "${REDIS_MASTER_PASSWORD}") ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf") ARGS+=("--include" "/opt/bitnami/redis/etc/replica.conf") /run.sh "${ARGS[@]}" env: - name: REDIS_REPLICATION_MODE value: slave - name: REDIS_MASTER_HOST value: redis-fixredis3-master-0.redis-fixredis3-headless.core-e.svc.cluster.local - name: REDIS_PORT value: "6379" - name: REDIS_MASTER_PORT_NUMBER value: "6379" - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-fixredis3 key: redis-password - name: REDIS_MASTER_PASSWORD valueFrom: secretKeyRef: name: redis-fixredis3 key: redis-password ports: - name: redis containerPort: 6379 livenessProbe: initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 exec: command: - sh - -c - /health/ping_liveness_local.sh 5 readinessProbe: initialDelaySeconds: 5 periodSeconds: 10 timeoutSeconds: 10 successThreshold: 1 failureThreshold: 5 exec: command: - sh - -c - /health/ping_readiness_local.sh 5 resources: null volumeMounts: - name: health mountPath: /health - name: redis-data mountPath: /data - name: config mountPath: /opt/bitnami/redis/mounted-etc - name: redis-tmp-conf mountPath: /opt/bitnami/redis/etc - name: sentinel image: cnc-fdn-docker.artifactory.prd.ss.aws.insim.biz/redis-sentinel:6.0.1-debian-10-r0 imagePullPolicy: "IfNotPresent" securityContext: runAsUser: 1001 command: - /bin/bash - -c - | if [[ -n $REDIS_PASSWORD_FILE ]]; then password_aux=`cat ${REDIS_PASSWORD_FILE}` export REDIS_PASSWORD=$password_aux fi if [[ ! -f /opt/bitnami/redis-sentinel/etc/sentinel.conf ]];then cp /opt/bitnami/redis-sentinel/mounted-etc/sentinel.conf /opt/bitnami/redis-sentinel/etc/sentinel.conf printf "\nsentinel auth-pass mymaster $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf printf "\nrequirepass $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf fi redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-fixredis3 key: redis-password - name: REDIS_SENTINEL_PORT value: "26379" ports: - name: redis-sentinel containerPort: 26379 livenessProbe: initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 exec: command: - sh - -c - /health/ping_sentinel.sh 5 readinessProbe: initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 5 exec: command: - sh - -c - /health/ping_sentinel.sh 5 resources: null volumeMounts: - name: health mountPath: /health - name: redis-data mountPath: /data subPath: - name: config mountPath: /opt/bitnami/redis-sentinel/mounted-etc - name: sentinel-tmp-conf mountPath: /opt/bitnami/redis-sentinel/etc - name: metrics image: cnc-fdn-docker.artifactory.prd.ss.aws.insim.biz/redis-exporter:1.6.0-debian-10-r2 imagePullPolicy: "IfNotPresent" command: - /bin/bash - -c - | if [[ -f '/secrets/redis-password' ]]; then export REDIS_PASSWORD=$(cat /secrets/redis-password) fi redis_exporter env: - name: REDIS_ALIAS value: redis-fixredis3 - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-fixredis3 key: redis-password volumeMounts: ports: - name: metrics containerPort: 9121 resources: null volumes: - name: health configMap: name: redis-fixredis3-health defaultMode: 0755 - name: config configMap: name: redis-fixredis3 - name: sentinel-tmp-conf emptyDir: {} - name: redis-tmp-conf emptyDir: {} volumeClaimTemplates: - metadata: name: redis-data labels: app: redis release: redis-fixredis3 heritage: Helm component: slave spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "8Gi" selector: updateStrategy: type: RollingUpdate ```

[ronaldvanderrest]

@ronaldvanderrest May be you can scale down the master stateful set after the chart installation and delete it.

Yes, this is now what we did for at least not having to worry about failing Nodes. However, its still cumbersome during updates, as it will than again reinitiate the master node, which will again start its own replication topology instead of using the existing one.

[andresbono]

Hi @ronaldvanderrest, we have done several tests, but we couldn't replicate the split brain situation that you described when the master node goes down. After a restart, the old master joins the cluster and becomes a slave of the new master.

About the old commit that you are referring to where the Redis master only has bootstrapping purposes, how would that solve the issue? Wouldn't the same problem appear with the new elected master?

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[ejpir]

@andresbono we have been able to narrow it down to calico networking and these 2 lines:

 existing_sentinels=$(timeout -s 9 5 redis-cli --raw -h testnick-redis -a "$REDIS_PASSWORD" -p 26379 SENTINEL sentinels mymaster)
 echo "$existing_sentinels" | awk -f /health/parse_sentinels.awk | tee -a /opt/bitnami/redis-sentinel/etc/sentinel.conf

will update the ticket when we know what part of calico is blocking the traffic. Without calico it works.

[andresbono]

Thank you very much for the information, @ejpir.

[ejpir]

we have been able to further narrow it down to a lost packet with only a SYN and no ACK.

Possible causes:

• Calico vs host iptables mismatch (1.6, v1.8, altho it seems calico runs in iptables-legacy mode, not nft).
• Snat
• Source/Destination checking on the ENI (AWS network interface). Nodes have more than one interface, might get trapped in one of those.

This simple workaround has got it working on our side, a simple retry, it alway succeeds on the 2nd try.

              # Somehow a packet gets lost, due to Calico, iptables or AWS K8s specifics. Eitherway, retrying fixes it.
              for i in {1..2}
                do
                  existing_sentinels=$(timeout -s 3 {{ .Values.sentinel.initialCheckTimeout }} redis-cli --raw -h {{ template "redis.fullname" . }} -a "$REDIS_PASSWORD" -p {{ .Values.sentinel.service.sentinelPort }} SENTINEL sentinels {{ .Values.sentinel.masterSet }})
                  echo "DEBUG sentinel: $existing_sentinels ..."
                  if [[ $existing_sentinels != "" ]]; then
                    echo "Found sentinal stopping.."
                    break
                  fi
                  echo "DEBUG: no sentinels found, retrying..."
                done

I hope I have more time and the needed expertise to debug it further and see where the packet is lost.

rejoin will show as:

redis-fixredis-master-0 sentinel DEBUG sentinel:  ...
redis-fixredis-master-0 sentinel DEBUG: no sentinels found, retrying...
redis-fixredis-master-0 sentinel Warning: AUTH failed
redis-fixredis-master-0 sentinel DEBUG sentinel: name <info>
redis-fixredis-master-0 sentinel 1:X 26 Jun 2020 14:27:38.009 * +sentinel-address-switch master mymaster 10.230.52.42 6379 ip 10.230.54.6 port 26379 for cc80a7e69fefa6ab30e8d7c4886e6e2fee43f0be
redis-fixredis-master-0 sentinel 1:X 26 Jun 2020 14:27:41.965 * +convert-to-slave slave 10.230.54.231:6379 10.230.54.231 6379 @ mymaster 10.230.52.42 6379
redis-fixredis-master-0 redis 1:S 26 Jun 2020 14:27:41.966 * REPLICAOF 10.230.52.42:6379 enabled (user request from 'id=8 addr=10.230.54.231:40481 fd=9 name=sentinel-22d19228-cmd age=10 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=200 qbuf-free=32568 obl=45 oll=0 omem=0 events=r cmd=exec user=default')

The AUTH failed is because we have no pass protection on sentinels. The chart somehow always has -a <password> as default. Removing -a or enabling pass did not fix it. It also doesn't make sense, since the retry mechanism is flawless in our case. Also played around with -r and -i, no luck there.

[andresbono]

Hi @ejpir, thanks for the detailed information. We are going to reopen the issue for some time in case someone else is also experiencing the same problem.

---

The AUTH failed is because we have no pass protection on sentinels. The chart somehow always has -a as default. Removing -a or enabling pass did not fix it. It also doesn't make sense, since the retry mechanism is flawless in our case. Also played around with -r and -i, no luck there.

I'm not sure if I'm understanding this part correctly. If you don't want to use password authentication, you should deploy the chart with usePassword: false and sentinel.usePassword: false. It should be configured in the first initialization to ensure it works.

[ejpir]

Hola @andresbono Yeah I understand the setting (I think ;P). I mean the default on this line:

• https://github.com/bitnami/charts/blob/bc8581861f8fa2bd0b2351f17efe7a329267ace5/bitnami/redis/templates/redis-master-statefulset.yaml#L245

The -a "$REDIS_PASSWORD" is always used, regardless of setting usePassword: false, right? shouldn't it be surrounded by an if? Or does the cli just ignore the parameter if the daemon has no password enabled/doesn't ask for it.

[andresbono]

@ejpir, adding an if there makes sense. Can you share the values that you are using to deploy the chart? We would like to reproduce the issue.

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[dimandzhi]

Hi,
I've had split-brain situation recently. I have 3 nodes with redis and sentinels (redis234, redis238, redis239). Servers had multiple power-down events after which redis239 and redis234 were found as masters. If I remember correctly, previously, redis239 was original master.

All mixed logs (redis+sentinel) left available:

redis239

redis_replicas.1.zli5ks34ymhzjgof8l5akzbqc /usr/lib/python3.11/site-packages/supervisor/options.py:474: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security. self.warnings.warn( 2024-01-20 16:55:32,237 INFO Set uid to user 0 succeeded 2024-01-20 16:55:32,245 INFO RPC interface 'supervisor' initialized 2024-01-20 16:55:32,245 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2024-01-20 16:55:32,245 INFO supervisord started with pid 7 2024-01-20 16:55:33,248 INFO spawned: 'process-stopper' with pid 10 2024-01-20 16:55:33,251 INFO spawned: 'log-redis' with pid 11 2024-01-20 16:55:33,254 INFO spawned: 'redis' with pid 12 2024-01-20 16:55:33,257 INFO spawned: 'log-sentinel' with pid 13 2024-01-20 16:55:33,259 INFO spawned: 'sentinel' with pid 15 Event listening started If any of processes stops, supervisor will be stopped 12:C 20 Jan 2024 16:55:33.278 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 12:C 20 Jan 2024 16:55:33.278 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=12, just started 12:C 20 Jan 2024 16:55:33.278 # Configuration loaded 12:S 20 Jan 2024 16:55:33.279 # Server initialized 12:S 20 Jan 2024 16:55:33.279 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 12:S 20 Jan 2024 16:55:33.832 # Done loading RDB, keys loaded: 7, keys expired: 0. 12:S 20 Jan 2024 16:55:34.237 # Unable to connect to MASTER: Invalid argument 2024-01-20 16:55:34,258 INFO success: process-stopper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 16:55:34,259 INFO success: log-redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 16:55:34,259 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 16:55:34,259 INFO success: log-sentinel entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 16:55:34,259 INFO success: sentinel entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 15:X 20 Jan 2024 16:55:33.277 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:33.278 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:33.278 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:33.279 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:33.279 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 15:X 20 Jan 2024 16:55:33.279 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=15, just started 15:X 20 Jan 2024 16:55:33.279 # Configuration loaded 15:X 20 Jan 2024 16:55:33.279 * monotonic clock: POSIX clock_gettime 15:X 20 Jan 2024 16:55:33.279 * Running mode=sentinel, port=26379. 15:X 20 Jan 2024 16:55:33.280 # Sentinel ID is db95d15ab390de46b7dda39b8195034c0d323f86 15:X 20 Jan 2024 16:55:33.280 # +monitor master mymaster redis234 6379 quorum 2 15:X 20 Jan 2024 16:55:33.282 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:33.282 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:33.283 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:33.283 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:33.754 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:33.784 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:35.242 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 16:55:34.326 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:34.327 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:34.328 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:34.329 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:36.248 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 16:55:35.373 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:35.374 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:35.375 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:35.376 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:35.775 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:35.832 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:35.890 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:35.891 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:37.254 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:36.421 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:36.422 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:36.423 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:36.424 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:37.451 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:37.452 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:37.453 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:37.454 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:37.813 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:37.890 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:37.937 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:37.938 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:38.259 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:38.480 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:38.481 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:38.482 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:38.483 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:39.264 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 16:55:39.499 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:39.500 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:39.501 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:39.502 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:39.849 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:39.902 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:39.963 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:39.964 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:40.269 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:40.544 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:40.545 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:40.547 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:40.548 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:41.274 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:41.617 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:41.618 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:41.620 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:41.621 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:41.874 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:41.930 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:41.967 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:41.968 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:42.281 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:42.690 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:42.691 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:42.692 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:42.693 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:43.286 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:43.691 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:43.692 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:43.693 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:43.694 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:43.883 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:43.950 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:43.987 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:43.988 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:44.290 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:44.719 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:44.720 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:44.721 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:44.722 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:45.296 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:45.778 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:45.779 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:45.780 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:45.781 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:45.934 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:46.025 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:46.026 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:46.027 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:46.302 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:46.827 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:46.828 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:46.829 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:46.830 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:47.306 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 16:55:47.865 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:47.866 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:47.867 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:47.868 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:47.983 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:48.042 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:48.083 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:48.084 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:48.312 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:48.880 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:48.881 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:48.882 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:48.883 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:49.318 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:49.944 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:49.945 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:49.946 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:49.948 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:50.031 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:50.084 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:50.097 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:50.098 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:50.323 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:50.949 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:50.950 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:50.951 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:50.952 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:51.327 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:51.957 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:51.958 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:51.959 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:51.960 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:52.106 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:52.107 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:52.151 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:52.152 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:52.334 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:53.002 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:53.004 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:53.005 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:53.006 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:53.339 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:54.023 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:54.024 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:54.025 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:54.026 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:54.117 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:54.118 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:54.170 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:54.171 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:54.345 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:55.061 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:55.062 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:55.063 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:55.064 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:55.349 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:56.085 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:56.086 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:56.087 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:56.088 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:56.124 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:56.125 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:56.217 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:56.218 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:56.355 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:57.145 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:57.146 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:57.146 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:57.147 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:57.360 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:58.133 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:58.134 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:58.184 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:58.185 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:58.186 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:58.187 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:58.257 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:58.258 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:55:58.366 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:55:59.202 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:55:59.203 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:59.204 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:55:59.205 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:55:59.370 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:56:00.196 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:00.196 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:00.224 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:00.225 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:00.226 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:00.227 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:56:00.376 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 16:56:00.280 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:00.282 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:01.261 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:01.262 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:01.263 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:01.264 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:56:01.381 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:56:02.224 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:02.225 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:56:02.387 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:56:02.300 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:02.302 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:02.303 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:02.304 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:02.305 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:02.306 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 16:56:03.392 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:56:03.326 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:03.326 # +sdown master mymaster redis234 6379 15:X 20 Jan 2024 16:56:03.327 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:03.327 # +sdown slave redis238:6379 redis238 6379 @ mymaster redis234 6379 15:X 20 Jan 2024 16:56:03.327 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:03.327 # +sdown sentinel a6b7e58db8d1b6c0d2cbb57fa14950e32225dbc4 redis238 26379 @ mymaster redis234 6379 15:X 20 Jan 2024 16:56:03.329 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:03.329 # +sdown sentinel 8dde94be3915f62c5ce429202ef4a54d3439a938 redis234 26379 @ mymaster redis234 6379 15:X 20 Jan 2024 16:56:04.257 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:04.257 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:56:04.403 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:56:04.336 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:04.337 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:04.389 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:04.390 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:04.391 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:04.391 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:56:05.435 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 16:56:05.468 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:05.468 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:05.469 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:05.470 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 16:56:06.368 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:06.369 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:06.483 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:06.483 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:06.485 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:06.486 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:06.536 # -sdown master mymaster redis234 6379 15:X 20 Jan 2024 16:56:06.536 # -sdown sentinel 8dde94be3915f62c5ce429202ef4a54d3439a938 redis234 26379 @ mymaster redis234 6379 15:X 20 Jan 2024 16:56:06.817 # Failed to resolve hostname 'redis234' 12:S 20 Jan 2024 16:56:07.797 # Done loading RDB, keys loaded: 7, keys expired: 0. 15:X 20 Jan 2024 16:56:07.510 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:07.510 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:56:08.658 # -sdown slave redis238:6379 redis238 6379 @ mymaster redis234 6379 15:X 20 Jan 2024 16:56:08.658 # -sdown sentinel a6b7e58db8d1b6c0d2cbb57fa14950e32225dbc4 redis238 26379 @ mymaster redis234 6379 15:X 20 Jan 2024 16:58:02.074 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:58:02.461 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:58:03.085 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:58:03.512 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:58:18.214 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:58:18.633 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:58:32.078 # +sdown sentinel a6b7e58db8d1b6c0d2cbb57fa14950e32225dbc4 redis238 26379 @ mymaster redis234 6379 15:X 20 Jan 2024 16:58:32.538 # +sdown slave redis238:6379 redis238 6379 @ mymaster redis234 6379 15:X 20 Jan 2024 16:58:33.322 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:58:33.769 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:58:48.440 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:58:48.881 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:03.564 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:04.032 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:06.625 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:06.627 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:09.715 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:09.716 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:12.817 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:12.819 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:15.866 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:15.867 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:18.962 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:18.963 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:22.008 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:22.010 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:25.085 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:25.086 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:28.126 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:28.128 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:31.286 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:31.288 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:34.310 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:34.311 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 16:59:38.445 * +reboot slave redis238:6379 redis238 6379 @ mymaster redis234 6379 15:X 20 Jan 2024 16:59:38.531 # -sdown slave redis238:6379 redis238 6379 @ mymaster redis234 6379 15:X 20 Jan 2024 16:59:38.531 # -sdown sentinel a6b7e58db8d1b6c0d2cbb57fa14950e32225dbc4 redis238 26379 @ mymaster redis234 6379 12:S 20 Jan 2024 17:28:04.550 # Connection with master lost. 12:S 20 Jan 2024 17:28:04.552 # Error condition on socket for SYNC: Connection refused 12:S 20 Jan 2024 17:28:04.767 # Error condition on socket for SYNC: Connection refused 15:X 20 Jan 2024 17:28:20.128 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:20.684 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:34.077 # +sdown master mymaster redis234 6379 15:X 20 Jan 2024 17:28:34.077 # +sdown sentinel 8dde94be3915f62c5ce429202ef4a54d3439a938 redis234 26379 @ mymaster redis234 6379 15:X 20 Jan 2024 17:28:34.351 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:34.427 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:34.431 # +new-epoch 18 15:X 20 Jan 2024 17:28:34.434 # +vote-for-leader a6b7e58db8d1b6c0d2cbb57fa14950e32225dbc4 18 15:X 20 Jan 2024 17:28:35.127 # +config-update-from sentinel a6b7e58db8d1b6c0d2cbb57fa14950e32225dbc4 redis238 26379 @ mymaster redis234 6379 15:X 20 Jan 2024 17:28:35.127 # +switch-master mymaster redis234 6379 redis239 6379 15:X 20 Jan 2024 17:28:35.128 * +slave slave redis238:6379 redis238 6379 @ mymaster redis239 6379 15:X 20 Jan 2024 17:28:35.129 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:35.129 * +slave slave redis234:6379 redis234 6379 @ mymaster redis239 6379 15:X 20 Jan 2024 17:28:35.173 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:35.228 # Failed to resolve hostname 'redis234' 12:M 20 Jan 2024 17:28:34.647 # Setting secondary replication ID to d1dcb441c503ae4cf1817367eb671c1a2e244dc5, valid up to offset: 4667990528. New replication ID is 41ad084371489c5785bc8c862a69ccfba920e8d0 12:M 20 Jan 2024 17:28:34.652 # CONFIG REWRITE executed with success. 15:X 20 Jan 2024 17:28:35.479 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:36.213 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:37.222 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:38.248 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:39.278 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:40.324 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:41.405 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:42.457 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:43.480 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:44.523 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:45.599 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:46.613 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:47.649 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:48.658 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:49.676 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:50.361 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:50.691 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:51.719 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:52.725 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:53.772 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:54.514 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:54.800 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:55.860 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:56.914 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:57.558 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:57.969 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:28:59.003 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:00.016 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:00.612 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:01.072 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:02.106 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:03.141 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:03.707 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:04.142 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:05.135 # +sdown slave redis234:6379 redis234 6379 @ mymaster redis239 6379 15:X 20 Jan 2024 17:29:05.203 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:06.228 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:06.757 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:07.275 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:08.292 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:09.298 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:09.867 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:10.343 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:11.373 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:12.412 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:12.956 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:13.436 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:14.476 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:15.541 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:15.998 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:16.595 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:17.644 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:18.680 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:19.099 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:19.697 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:20.752 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:21.814 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:22.113 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:22.844 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:23.846 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:24.889 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:25.185 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:25.899 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:26.931 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:27.951 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:28.251 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:28.981 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:30.035 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:31.080 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:31.327 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:32.105 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:33.104 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:34.172 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:34.410 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:35.191 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:36.217 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:37.249 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:37.536 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:38.304 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:39.353 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:40.386 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:40.554 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:41.407 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:42.442 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:43.494 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:43.645 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:44.529 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:45.534 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:46.569 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:46.706 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:47.586 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:48.609 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:49.634 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:49.774 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:50.657 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:51.695 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:52.705 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:52.859 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:53.734 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:54.792 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:55.835 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:55.937 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:56.862 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:57.870 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:58.923 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:58.977 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:29:59.939 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:00.955 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:02.029 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:02.085 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:03.034 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:04.084 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:05.090 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:05.149 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:06.106 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:07.132 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:08.211 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:08.212 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:09.300 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:10.328 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:11.263 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:11.392 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:12.445 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:13.496 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:14.348 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:14.527 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:15.584 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:16.610 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:17.411 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:17.660 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:18.711 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:19.784 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:20.505 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:20.823 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:21.839 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:22.862 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:23.600 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:23.864 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:24.933 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:25.974 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:26.656 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:27.062 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:28.106 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:29.135 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:29.719 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:30.179 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:31.201 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:32.240 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:32.820 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:33.292 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:34.316 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:35.339 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:35.859 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:36.362 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:37.390 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:38.452 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:38.924 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:39.489 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:40.529 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:41.543 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:42.012 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:42.637 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:43.659 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:44.746 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:45.053 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:45.751 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:46.779 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:47.807 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:48.155 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:48.854 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:49.862 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:50.874 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:51.240 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:51.921 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:52.973 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:54.015 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:54.283 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:55.042 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:56.106 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:57.115 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:57.383 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:58.142 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:30:59.163 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:00.225 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:00.470 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:01.251 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:02.252 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:03.308 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:03.518 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:04.337 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:05.389 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:06.477 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:06.568 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:07.491 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:08.525 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:09.567 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:09.631 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:10.639 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:11.672 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:12.690 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:12.782 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:13.692 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:14.752 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:15.806 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:15.808 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:16.819 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:17.859 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:18.904 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:18.905 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:19.927 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:20.942 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:21.934 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:22.007 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:23.014 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:24.023 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:25.051 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:25.052 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:26.053 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:27.130 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:28.103 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:28.156 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:29.175 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:30.186 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:31.177 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:31.240 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:32.242 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:33.295 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:34.234 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:34.298 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:35.310 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:36.321 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:37.296 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:37.381 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:38.419 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:39.469 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:40.383 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:40.519 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:41.568 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:42.621 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:43.481 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:43.680 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:44.707 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:45.764 # Failed to resolve hostname 'redis234' 15:X 20 Jan 2024 17:31:47.651 # -sdown sentinel 8dde94be3915f62c5ce429202ef4a54d3439a938 redis234 26379 @ mymaster redis239 6379 15:X 20 Jan 2024 17:31:47.964 # -sdown slave redis234:6379 redis234 6379 @ mymaster redis239 6379 12:M 21 Jan 2024 13:27:44.141 # Connection with replica redis234:6379 lost. 15:X 21 Jan 2024 13:27:45.186 # Failed to resolve hostname 'redis234' 15:X 21 Jan 2024 13:27:45.187 # Failed to resolve hostname 'redis234' 15:X 21 Jan 2024 13:28:00.291 * +reboot slave redis234:6379 redis234 6379 @ mymaster redis239 6379

redis234

redis_replicas.2.z678jhftp8xg6p44sr8kgpa97 /usr/lib/python3.11/site-packages/supervisor/options.py:474: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security. self.warnings.warn( 2024-01-20 17:31:45,984 INFO Set uid to user 0 succeeded 2024-01-20 17:31:45,989 INFO RPC interface 'supervisor' initialized 2024-01-20 17:31:45,989 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2024-01-20 17:31:45,989 INFO supervisord started with pid 7 2024-01-20 17:31:46,992 INFO spawned: 'process-stopper' with pid 10 2024-01-20 17:31:46,995 INFO spawned: 'log-redis' with pid 11 2024-01-20 17:31:46,997 INFO spawned: 'redis' with pid 12 2024-01-20 17:31:46,999 INFO spawned: 'log-sentinel' with pid 13 2024-01-20 17:31:47,002 INFO spawned: 'sentinel' with pid 15 Event listening started If any of processes stops, supervisor will be stopped 12:C 20 Jan 2024 17:31:47.015 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 12:C 20 Jan 2024 17:31:47.015 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=12, just started 12:C 20 Jan 2024 17:31:47.015 # Configuration loaded 12:M 20 Jan 2024 17:31:47.016 # Server initialized 12:M 20 Jan 2024 17:31:47.016 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 12:M 20 Jan 2024 17:31:47.101 # Done loading RDB, keys loaded: 9, keys expired: 0. 2024-01-20 17:31:48,002 INFO success: process-stopper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 17:31:48,003 INFO success: log-redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 17:31:48,003 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 17:31:48,003 INFO success: log-sentinel entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 17:31:48,003 INFO success: sentinel entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 15:X 20 Jan 2024 17:31:47.032 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:47.033 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:47.050 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:47.050 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:47.051 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 15:X 20 Jan 2024 17:31:47.051 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=15, just started 15:X 20 Jan 2024 17:31:47.051 # Configuration loaded 15:X 20 Jan 2024 17:31:47.051 * monotonic clock: POSIX clock_gettime 15:X 20 Jan 2024 17:31:47.052 * Running mode=sentinel, port=26379. 15:X 20 Jan 2024 17:31:47.053 # Sentinel ID is 8dde94be3915f62c5ce429202ef4a54d3439a938 15:X 20 Jan 2024 17:31:47.053 # +monitor master mymaster redis234 6379 quorum 2 15:X 20 Jan 2024 17:31:47.054 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:47.055 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:47.056 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:47.056 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:47.617 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:47.623 # +new-epoch 18 15:X 20 Jan 2024 17:31:47.624 # +config-update-from sentinel db95d15ab390de46b7dda39b8195034c0d323f86 redis239 26379 @ mymaster redis234 6379 15:X 20 Jan 2024 17:31:47.624 # +switch-master mymaster redis234 6379 redis239 6379 15:X 20 Jan 2024 17:31:47.625 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:47.707 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:47.711 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:47.911 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:48.119 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:48.119 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:48.120 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:48.121 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:49.176 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:49.177 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:49.178 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:49.179 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:49.676 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:49.728 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:49.729 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:49.957 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:50.199 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:50.200 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:50.204 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:50.205 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:51.245 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:51.246 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:51.247 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:51.248 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:51.730 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:51.744 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:51.745 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:52.008 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:52.286 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:52.287 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:52.288 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:52.289 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:53.320 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:53.321 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:53.322 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:53.323 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:53.739 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:53.796 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:53.797 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:54.031 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:54.320 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:54.321 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:54.322 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:54.322 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:55.355 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:55.356 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:55.357 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:55.358 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:55.782 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:55.850 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:55.851 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:56.113 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:56.409 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:56.410 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:56.411 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:56.412 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:31:57.745 # Unable to connect to MASTER: Invalid argument 12:S 20 Jan 2024 17:31:57.750 # CONFIG REWRITE executed with success. 15:X 20 Jan 2024 17:31:57.449 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:57.450 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:57.451 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:57.452 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:57.817 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:57.864 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:57.865 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:31:58.010 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 17:31:58.136 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:58.466 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:58.467 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:58.468 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:58.468 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:31:59.016 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:31:59.482 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:59.483 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:59.483 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:59.484 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:59.842 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:31:59.874 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:31:59.875 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:00.021 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:00.172 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:00.580 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:00.581 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:00.582 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:00.583 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:01.026 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:01.649 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:01.650 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:01.651 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:01.651 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:01.901 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:01.928 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:01.930 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:02.029 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 17:32:02.215 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:02.652 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:02.653 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:02.654 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:02.655 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:03.032 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:03.706 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:03.707 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:03.708 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:03.708 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:03.910 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:03.933 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:03.934 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:04.035 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 17:32:04.280 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:04.730 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:04.731 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:04.732 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:04.733 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:05.038 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:05.768 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:05.769 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:05.770 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:05.771 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:05.924 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:05.979 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:05.980 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:06.040 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:06.306 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:06.783 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:06.784 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:06.784 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:06.785 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:07.043 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:07.788 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:07.789 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:07.790 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:07.791 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:07.971 # Failed to resolve hostname 'redis239' 12:S 20 Jan 2024 17:32:08.046 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 17:32:08.043 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:08.044 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:08.319 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:08.842 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:08.843 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:08.844 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:08.845 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:09.051 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:09.862 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:09.863 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:09.864 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:09.864 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:10.057 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:10.016 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:10.098 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:10.099 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:10.356 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:10.909 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:10.910 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:10.911 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:10.912 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:11.063 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:11.911 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:11.912 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:11.913 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:11.914 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:12.070 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 17:32:12.022 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:12.159 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:12.160 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:12.419 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:12.934 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:12.935 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:12.936 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:12.936 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:13.077 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:13.945 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:13.946 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:13.946 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:13.947 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:14.083 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:14.048 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:14.167 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:14.168 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:14.450 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:14.950 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:14.951 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:14.952 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:14.952 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:15.087 # Unable to connect to MASTER: Invalid argument 15:X 20 Jan 2024 17:32:15.954 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:15.954 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:15.955 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:15.956 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:16.093 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 17:32:16.065 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:16.201 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:16.203 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:16.520 # Failed to resolve hostname 'redis239' 12:S 20 Jan 2024 17:32:17.098 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 17:32:17.020 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:17.021 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:17.022 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:17.022 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:17.081 # +sdown slave redis238:6379 redis238 6379 @ mymaster redis234 6379 15:X 20 Jan 2024 17:32:17.081 # +sdown slave redis239:6379 redis239 6379 @ mymaster redis234 6379 15:X 20 Jan 2024 17:32:17.081 # +sdown sentinel db95d15ab390de46b7dda39b8195034c0d323f86 redis239 26379 @ mymaster redis234 6379 15:X 20 Jan 2024 17:32:17.081 # +sdown sentinel a6b7e58db8d1b6c0d2cbb57fa14950e32225dbc4 redis238 26379 @ mymaster redis234 6379 12:S 20 Jan 2024 17:32:18.104 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 17:32:18.029 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:18.029 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:18.030 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:18.031 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:18.108 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:18.208 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:18.209 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:18.530 # Failed to resolve hostname 'redis239' 12:S 20 Jan 2024 17:32:19.109 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 17:32:19.049 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:19.050 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:19.051 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:19.051 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:20.114 # Unable to connect to MASTER: Resource temporarily unavailable 15:X 20 Jan 2024 17:32:20.078 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:20.079 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:20.079 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:20.080 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:20.185 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:20.216 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:20.216 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:20.562 # Failed to resolve hostname 'redis239' 15:X 20 Jan 2024 17:32:21.105 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:21.107 # Failed to resolve hostname 'redis238' 15:X 20 Jan 2024 17:32:21.183 # -sdown slave redis239:6379 redis239 6379 @ mymaster redis234 6379 15:X 20 Jan 2024 17:32:21.183 # -sdown sentinel db95d15ab390de46b7dda39b8195034c0d323f86 redis239 26379 @ mymaster redis234 6379 15:X 20 Jan 2024 17:32:21.236 # Failed to resolve hostname 'redis238' 12:S 20 Jan 2024 17:32:22.462 # Done loading RDB, keys loaded: 7, keys expired: 0. 15:X 20 Jan 2024 17:32:22.188 # -sdown slave redis238:6379 redis238 6379 @ mymaster redis234 6379 15:X 20 Jan 2024 17:32:22.188 # -sdown sentinel a6b7e58db8d1b6c0d2cbb57fa14950e32225dbc4 redis238 26379 @ mymaster redis234 6379 15:X 20 Jan 2024 17:32:57.149 # +sdown master mymaster redis234 6379 2024-01-21 13:27:44,062 INFO waiting for sentinel to stop 2024-01-21 13:27:44,067 INFO stopped: sentinel (exit status 0) Headers: "{'ver': '3.0', 'server': 'supervisor', 'serial': '0', 'pool': 'process-stopper', 'poolserial': '0', 'eventname': 'PROCESS_STATE_STOPPED', 'len': '66'}" Body: "{'processname': 'sentinel', 'groupname': 'sentinel', 'from_state': 'STOPPING', 'pid': '15'}" Stopping supervisord 2024-01-21 13:27:44,067 WARN received SIGQUIT indicating exit request 2024-01-21 13:27:44,067 INFO waiting for process-stopper, log-redis, redis, log-sentinel to die 2024-01-21 13:27:44,068 WARN stopped: log-sentinel (terminated by SIGTERM) 2024-01-21 13:27:44,083 INFO exited: process-stopper (exit status 0; expected) 2024-01-21 13:27:44,142 INFO stopped: redis (exit status 0) 2024-01-21 13:27:44,143 WARN stopped: log-redis (terminated by SIGTERM) *** REBOOT *** redis_replicas.2.mgbikgnqx5esa7225axbig0od /usr/lib/python3.11/site-packages/supervisor/options.py:474: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security. self.warnings.warn( 2024-01-21 13:27:51,353 INFO Set uid to user 0 succeeded 2024-01-21 13:27:51,356 INFO RPC interface 'supervisor' initialized 2024-01-21 13:27:51,356 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2024-01-21 13:27:51,356 INFO supervisord started with pid 7 2024-01-21 13:27:52,359 INFO spawned: 'process-stopper' with pid 10 2024-01-21 13:27:52,362 INFO spawned: 'log-redis' with pid 11 2024-01-21 13:27:52,364 INFO spawned: 'redis' with pid 12 2024-01-21 13:27:52,366 INFO spawned: 'log-sentinel' with pid 13 2024-01-21 13:27:52,368 INFO spawned: 'sentinel' with pid 15 Event listening started If any of processes stops, supervisor will be stopped 12:C 21 Jan 2024 13:27:52.377 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 12:C 21 Jan 2024 13:27:52.377 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=12, just started 12:C 21 Jan 2024 13:27:52.377 # Configuration loaded 12:S 21 Jan 2024 13:27:52.379 # Server initialized 12:S 21 Jan 2024 13:27:52.379 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 12:S 21 Jan 2024 13:27:52.765 # Done loading RDB, keys loaded: 7, keys expired: 0. 2024-01-21 13:27:53,369 INFO success: process-stopper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-21 13:27:53,369 INFO success: log-redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-21 13:27:53,369 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-21 13:27:53,369 INFO success: log-sentinel entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-21 13:27:53,369 INFO success: sentinel entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 15:X 21 Jan 2024 13:27:52.383 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 15:X 21 Jan 2024 13:27:52.383 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=15, just started 15:X 21 Jan 2024 13:27:52.383 # Configuration loaded 15:X 21 Jan 2024 13:27:52.384 * monotonic clock: POSIX clock_gettime 15:X 21 Jan 2024 13:27:52.384 * Running mode=sentinel, port=26379. 15:X 21 Jan 2024 13:27:52.385 # Sentinel ID is 8dde94be3915f62c5ce429202ef4a54d3439a938 15:X 21 Jan 2024 13:27:52.385 # +monitor master mymaster redis234 6379 quorum 2 12:S 21 Jan 2024 13:27:53.499 # Done loading RDB, keys loaded: 2, keys expired: 0. 15:X 21 Jan 2024 13:27:53.468 # +config-update-from sentinel db95d15ab390de46b7dda39b8195034c0d323f86 redis239 26379 @ mymaster redis234 6379 15:X 21 Jan 2024 13:27:53.468 # +switch-master mymaster redis234 6379 redis239 6379 15:X 21 Jan 2024 13:27:53.469 * +slave slave redis238:6379 redis238 6379 @ mymaster redis239 6379 15:X 21 Jan 2024 13:27:53.469 * +slave slave redis234:6379 redis234 6379 @ mymaster redis239 6379

redis238

redis_replicas.3.ae3ioo6ald9kf0v9ex4sv4z4j /usr/lib/python3.11/site-packages/supervisor/options.py:474: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security. self.warnings.warn( 2024-01-20 16:59:36,862 INFO Set uid to user 0 succeeded 2024-01-20 16:59:36,865 INFO RPC interface 'supervisor' initialized 2024-01-20 16:59:36,865 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2024-01-20 16:59:36,865 INFO supervisord started with pid 7 2024-01-20 16:59:37,867 INFO spawned: 'process-stopper' with pid 10 2024-01-20 16:59:37,868 INFO spawned: 'log-redis' with pid 11 2024-01-20 16:59:37,870 INFO spawned: 'redis' with pid 12 2024-01-20 16:59:37,872 INFO spawned: 'log-sentinel' with pid 13 2024-01-20 16:59:37,874 INFO spawned: 'sentinel' with pid 16 Event listening started If any of processes stops, supervisor will be stopped 12:C 20 Jan 2024 16:59:37.879 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 12:C 20 Jan 2024 16:59:37.879 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=12, just started 12:C 20 Jan 2024 16:59:37.879 # Configuration loaded 12:S 20 Jan 2024 16:59:37.880 # Server initialized 12:S 20 Jan 2024 16:59:37.880 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 12:S 20 Jan 2024 16:59:38.491 # Done loading RDB, keys loaded: 7, keys expired: 0. 2024-01-20 16:59:38,874 INFO success: process-stopper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 16:59:38,875 INFO success: log-redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 16:59:38,875 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 16:59:38,875 INFO success: log-sentinel entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-01-20 16:59:38,875 INFO success: sentinel entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 16:X 20 Jan 2024 16:59:37.881 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 16:X 20 Jan 2024 16:59:37.881 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=16, just started 16:X 20 Jan 2024 16:59:37.881 # Configuration loaded 16:X 20 Jan 2024 16:59:37.882 * monotonic clock: POSIX clock_gettime 16:X 20 Jan 2024 16:59:37.882 * Running mode=sentinel, port=26379. 16:X 20 Jan 2024 16:59:37.882 # Sentinel ID is a6b7e58db8d1b6c0d2cbb57fa14950e32225dbc4 16:X 20 Jan 2024 16:59:37.882 # +monitor master mymaster redis234 6379 quorum 2 12:S 20 Jan 2024 16:59:40.088 # Done loading RDB, keys loaded: 7, keys expired: 0. 12:S 20 Jan 2024 17:28:04.576 # Connection with master lost. 12:S 20 Jan 2024 17:28:04.584 # Error condition on socket for SYNC: Connection refused 16:X 20 Jan 2024 17:28:05.731 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:20.262 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:20.854 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:34.104 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:34.117 # +sdown sentinel 8dde94be3915f62c5ce429202ef4a54d3439a938 redis234 26379 @ mymaster redis234 6379 16:X 20 Jan 2024 17:28:34.375 # +sdown master mymaster redis234 6379 16:X 20 Jan 2024 17:28:34.447 # +odown master mymaster redis234 6379 #quorum 2/2 16:X 20 Jan 2024 17:28:34.447 # +new-epoch 18 16:X 20 Jan 2024 17:28:34.447 # +try-failover master mymaster redis234 6379 16:X 20 Jan 2024 17:28:34.451 # +vote-for-leader a6b7e58db8d1b6c0d2cbb57fa14950e32225dbc4 18 16:X 20 Jan 2024 17:28:34.461 # db95d15ab390de46b7dda39b8195034c0d323f86 voted for a6b7e58db8d1b6c0d2cbb57fa14950e32225dbc4 18 16:X 20 Jan 2024 17:28:34.542 # +elected-leader master mymaster redis234 6379 16:X 20 Jan 2024 17:28:34.542 # +failover-state-select-slave master mymaster redis234 6379 16:X 20 Jan 2024 17:28:34.613 # +selected-slave slave redis239:6379 redis239 6379 @ mymaster redis234 6379 16:X 20 Jan 2024 17:28:34.613 * +failover-state-send-slaveof-noone slave redis239:6379 redis239 6379 @ mymaster redis234 6379 16:X 20 Jan 2024 17:28:34.672 * +failover-state-wait-promotion slave redis239:6379 redis239 6379 @ mymaster redis234 6379 16:X 20 Jan 2024 17:28:35.056 # +promoted-slave slave redis239:6379 redis239 6379 @ mymaster redis234 6379 16:X 20 Jan 2024 17:28:35.056 # +failover-state-reconf-slaves master mymaster redis234 6379 12:S 20 Jan 2024 17:28:35.160 # CONFIG REWRITE executed with success. 12:S 20 Jan 2024 17:28:35.161 # Master replication ID changed to 41ad084371489c5785bc8c862a69ccfba920e8d0 16:X 20 Jan 2024 17:28:35.152 * +slave-reconf-sent slave redis238:6379 redis238 6379 @ mymaster redis234 6379 16:X 20 Jan 2024 17:28:35.367 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:35.567 # -odown master mymaster redis234 6379 16:X 20 Jan 2024 17:28:35.955 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:35.956 * +slave-reconf-inprog slave redis238:6379 redis238 6379 @ mymaster redis234 6379 16:X 20 Jan 2024 17:28:35.956 * +slave-reconf-done slave redis238:6379 redis238 6379 @ mymaster redis234 6379 16:X 20 Jan 2024 17:28:36.011 # +failover-end master mymaster redis234 6379 16:X 20 Jan 2024 17:28:36.011 # +switch-master mymaster redis234 6379 redis239 6379 16:X 20 Jan 2024 17:28:36.011 * +slave slave redis238:6379 redis238 6379 @ mymaster redis239 6379 16:X 20 Jan 2024 17:28:36.012 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:36.013 * +slave slave redis234:6379 redis234 6379 @ mymaster redis239 6379 16:X 20 Jan 2024 17:28:36.076 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:37.125 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:38.153 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:39.209 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:40.253 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:41.275 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:42.306 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:43.350 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:44.381 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:45.423 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:46.435 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:47.476 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:48.484 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:49.509 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:50.482 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:50.539 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:51.587 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:52.590 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:53.671 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:54.606 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:54.732 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:55.772 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:56.810 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:57.676 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:57.849 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:58.881 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:28:59.927 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:00.751 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:00.949 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:01.971 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:03.009 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:03.800 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:04.027 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:05.065 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:06.035 # +sdown slave redis234:6379 redis234 6379 @ mymaster redis239 6379 16:X 20 Jan 2024 17:29:06.127 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:06.938 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:07.138 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:08.182 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:09.195 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:09.950 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:10.257 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:11.344 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:12.351 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:13.001 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:13.446 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:14.468 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:15.502 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:16.116 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:16.527 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:17.528 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:18.553 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:19.203 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:19.620 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:20.685 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:21.707 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:22.247 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:22.713 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:23.754 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:24.771 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:25.353 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:25.810 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:26.817 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:27.826 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:28.399 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:28.877 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:29.884 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:30.911 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:31.448 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:31.968 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:33.011 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:34.060 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:34.513 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:35.130 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:36.152 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:37.169 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:37.595 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:38.194 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:39.213 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:40.300 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:40.694 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:41.332 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:42.358 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:43.374 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:43.717 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:44.375 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:45.380 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:46.427 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:46.856 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:47.491 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:48.494 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:49.505 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:49.865 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:50.526 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:51.615 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:52.645 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:52.961 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:53.744 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:54.799 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:55.835 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:56.042 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:56.861 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:57.918 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:58.978 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:29:59.118 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:00.013 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:01.013 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:02.060 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:02.189 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:03.121 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:04.134 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:05.138 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:05.283 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:06.180 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:07.276 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:08.298 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:08.299 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:09.328 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:10.397 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:11.383 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:11.440 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:12.460 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:13.473 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:14.490 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:14.491 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:15.506 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:16.536 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:17.538 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:17.539 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:18.584 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:19.597 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:20.582 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:20.645 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:21.667 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:22.701 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:23.688 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:23.744 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:24.807 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:25.865 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:26.782 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:26.940 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:27.996 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:29.080 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:29.804 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:30.119 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:31.154 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:32.208 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:32.910 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:33.270 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:34.299 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:35.341 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:36.005 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:36.351 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:37.375 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:38.428 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:39.036 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:39.472 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:40.487 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:41.509 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:42.165 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:42.525 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:43.554 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:44.585 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:45.166 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:45.619 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:46.639 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:47.675 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:48.228 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:48.726 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:49.788 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:50.842 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:51.325 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:51.856 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:52.902 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:53.952 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:54.407 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:55.051 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:56.061 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:57.091 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:57.475 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:58.107 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:30:59.148 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:00.153 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:00.576 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:01.181 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:02.191 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:03.232 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:03.645 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:04.239 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:05.266 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:06.309 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:06.675 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:07.330 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:08.346 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:09.399 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:09.739 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:10.438 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:11.508 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:12.523 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:12.820 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:13.548 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:14.566 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:15.597 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:15.871 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:16.652 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:17.703 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:18.732 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:18.977 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:19.744 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:20.780 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:21.801 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:22.036 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:22.814 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:23.855 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:24.876 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:25.144 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:25.896 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:26.928 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:27.940 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:28.200 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:28.973 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:30.008 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:31.026 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:31.247 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:32.069 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:33.103 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:34.176 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:34.345 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:35.227 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:36.237 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:37.254 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:37.385 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:38.272 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:39.317 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:40.327 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:40.496 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:41.396 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:42.447 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:43.498 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:43.558 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:44.554 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:45.594 # Failed to resolve hostname 'redis234' 16:X 20 Jan 2024 17:31:47.763 # -sdown slave redis234:6379 redis234 6379 @ mymaster redis239 6379 16:X 20 Jan 2024 17:31:47.763 # -sdown sentinel 8dde94be3915f62c5ce429202ef4a54d3439a938 redis234 26379 @ mymaster redis239 6379 16:X 20 Jan 2024 17:31:57.739 * +convert-to-slave slave redis234:6379 redis234 6379 @ mymaster redis239 6379 16:X 21 Jan 2024 13:27:45.153 # Failed to resolve hostname 'redis234' 16:X 21 Jan 2024 13:27:45.244 # Failed to resolve hostname 'redis234' 16:X 21 Jan 2024 13:28:00.366 * +reboot slave redis234:6379 redis234 6379 @ mymaster redis239 6379

I did use REPLICAOF command at some point to solve situation.