search

bitnami / containers

Bitnami container images
https://bitnami.com
Other
3.03k stars 4.38k forks source link

[bitnami/redis-cluster] nodes never ready if using REDIS_PASSWORD_FILE instead of REDIS_PASSWORD #35982

Open byErikas opened 1 year ago

byErikas commented 1 year ago

Name and Version

bitnami/redis-cluster:7.0

What architecture are you using?

amd64

What steps will reproduce the bug?

I'm trying to run a docker stack on a swarm enviroment for the redis cluster, this is how the docker-compose looks: 

version: '3.8'

services:
## REDIS
  redis-node-0:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-0:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

  redis-node-1:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-1:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

  redis-node-2:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-2:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

  redis-node-3:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-3:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

  redis-node-4:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-4:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

  redis-node-5:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-5:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDISCLI_AUTH_FILE: /run/secrets/redis_password
      REDIS_CLUSTER_CREATOR: "yes"
      REDIS_CLUSTER_REPLICAS: 1
      REDIS_CLUSTER_SLEEP_BEFORE_DNS_LOOKUP: 30
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
      BITNAMI_DEBUG: 1
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

## END OF SERVICES ----------------------
## CONFIGS / SECRETS / VOLUMES / NETWORKS
configs:
  redis_conf:
    external: true
    name: web_vast-redis_conf

secrets:
  redis_password:
    external: true
    name: web_vast-redis_password

volumes:
  volume-redis-0:
    name: web_vast-redis-0
  volume-redis-1:
    name: web_vast-redis-1
  volume-redis-2:
    name: web_vast-redis-2
  volume-redis-3:
    name: web_vast-redis-3
  volume-redis-4:
    name: web_vast-redis-4
  volume-redis-5:
    name: web_vast-redis-5

networks:
  network-stack:
    driver: overlay
    name: web_vast-local

I've been trying to follow the setup based on the README.md over here: https://github.com/bitnami/containers/blob/main/bitnami/redis-cluster/README.md, however I need the redis password to be a secret that gets mounted on the container for redis, and I didn't see anything related to being able to use files for variables in the readme, so I went to dig around, and found that there seems to be some support for the "_FILE" variables over in this piece of the initial code https://github.com/bitnami/containers/blob/main/bitnami/redis-cluster/7.0/debian-11/rootfs/opt/bitnami/scripts/redis-cluster-env.sh#L68, however this doesn't seem to work.

I know that the "_FILE" support isn't in the documentation, and is likely not supported, bus is there any way I could set it up to read the password from a docker secret?

What is the expected behavior?

I had hopes that the cluster would start and initiate the same way it does if you simply use REDIS_PASSWORD in the docker-compose, as it works without any issues when configured using that, however it doesn't.

What do you see instead?

What seems to happen is that either the nodes, or the cluster are never ready, looking at the logs for the container that I use to initialize the cluster I get this:

redis-cluster 10:23:12.17 

redis-cluster 10:23:12.17 Welcome to the Bitnami redis-cluster container

redis-cluster 10:23:12.18 Subscribe to project updates by watching https://github.com/bitnami/containers

redis-cluster 10:23:12.18 Submit issues and feature requests at https://github.com/bitnami/containers/issues

redis-cluster 10:23:12.18 

redis-cluster 10:23:12.18 INFO  ==> ** Starting Redis setup **

redis-cluster 10:23:12.21 DEBUG ==> Validating settings in REDIS_* env vars..

redis-cluster 10:23:12.21 INFO  ==> Initializing Redis

redis-cluster 10:23:12.22 DEBUG ==> Ensuring expected directories/files exist

redis-cluster 10:23:12.24 INFO  ==> Setting Redis config file

Storing map with hostnames and IPs

redis-cluster 10:23:27.48 INFO  ==> ** Redis setup finished! **

140:C 31 May 2023 10:23:27.526 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

140:C 31 May 2023 10:23:27.526 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=140, just started

140:C 31 May 2023 10:23:27.526 # Configuration loaded

140:M 31 May 2023 10:23:27.527 * monotonic clock: POSIX clock_gettime

Node redis-node-0 not ready, waiting for all the nodes to be ready...

140:M 31 May 2023 10:23:27.528 * No cluster configuration found, I'm 2511621d9161c971e47c4317a5cea80ec8daed66

140:M 31 May 2023 10:23:27.530 * Running mode=cluster, port=6379.

140:M 31 May 2023 10:23:27.530 # Server initialized

140:M 31 May 2023 10:23:27.530 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

140:M 31 May 2023 10:23:27.531 * Creating AOF base file appendonly.aof.1.base.rdb on server start

140:M 31 May 2023 10:23:27.531 * Creating AOF incr file appendonly.aof.1.incr.aof on server start

140:M 31 May 2023 10:23:27.532 * Ready to accept connections

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

And it just keeps going forever, while if I inspect the node named redis-node-0 I see this:

redis-cluster 10:24:19.52 

redis-cluster 10:24:19.52 Welcome to the Bitnami redis-cluster container

redis-cluster 10:24:19.52 Subscribe to project updates by watching https://github.com/bitnami/containers

redis-cluster 10:24:19.53 Submit issues and feature requests at https://github.com/bitnami/containers/issues

redis-cluster 10:24:19.53 

redis-cluster 10:24:19.53 INFO  ==> ** Starting Redis setup **

redis-cluster 10:24:19.56 INFO  ==> Initializing Redis

redis-cluster 10:24:19.58 INFO  ==> Setting Redis config file

Changing old IP 10.0.82.21 by the new one 10.0.82.21

Changing old IP 10.0.82.13 by the new one 10.0.82.13

Changing old IP 10.0.82.17 by the new one 10.0.82.17

Changing old IP 10.0.82.19 by the new one 10.0.82.19

Changing old IP 10.0.82.23 by the new one 10.0.82.23

Changing old IP 10.0.82.9 by the new one 10.0.82.9

redis-cluster 10:24:19.77 INFO  ==> ** Redis setup finished! **

1:C 31 May 2023 10:24:19.813 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

1:C 31 May 2023 10:24:19.813 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=1, just started

1:C 31 May 2023 10:24:19.813 # Configuration loaded

1:M 31 May 2023 10:24:19.814 * monotonic clock: POSIX clock_gettime

1:M 31 May 2023 10:24:19.814 * Node configuration loaded, I'm 6ef7ffa00ff1632636efee1d17426d53eeacf111

1:M 31 May 2023 10:24:19.815 * Running mode=cluster, port=6379.

1:M 31 May 2023 10:24:19.815 # Server initialized

1:M 31 May 2023 10:24:19.815 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

1:M 31 May 2023 10:24:19.816 * Reading RDB base file on AOF loading...

1:M 31 May 2023 10:24:19.816 * Loading RDB produced by version 7.0.11

1:M 31 May 2023 10:24:19.816 * RDB age 53 seconds

1:M 31 May 2023 10:24:19.816 * RDB memory usage when created 1.51 Mb

1:M 31 May 2023 10:24:19.816 * RDB is base AOF

1:M 31 May 2023 10:24:19.816 * Done loading RDB, keys loaded: 0, keys expired: 0.

1:M 31 May 2023 10:24:19.816 * DB loaded from base file appendonly.aof.1.base.rdb: 0.000 seconds

1:M 31 May 2023 10:24:19.816 * DB loaded from append only file: 0.000 seconds

1:M 31 May 2023 10:24:19.816 * Opening AOF incr file appendonly.aof.1.incr.aof on server start

1:M 31 May 2023 10:24:19.816 * Ready to accept connections

Additional information

The only thing that is in my redis_conf that gets put into overrides.conf is this:

maxmemory-policy volatile-lru

so I don't think that this is a source of any issues

byErikas commented 1 year ago

I've managed to narrow it down to a problem with the REDISCLI_AUTH variable, please take a look at https://github.com/bitnami/containers/pull/35992 for a fix that seems to work atleast for my enviroment.

carrodher commented 1 year ago

Thanks for creating this issue and the associated PR. The team will review it and provide feedback. Once merged the PR, this issue will be automatically closed.

joancafom commented 1 year ago

Hi @byErikas

I've recently reverted the changes introduced to support REDISCLI_AUTH from a file (see https://github.com/bitnami/containers/pull/39867) due to them causing some problems when redis was configured without a password.

I've created an internal task to evaluate the feasibility of this feature and the best way to tackle it, so we'll keep the thread updated when we have more news.

davama commented 14 hours ago

Good Day,

Dont mean to hijack this post but trying something similar to what the OP is doing.

Trying to understand how to set a password when ALLOW_EMPTY_PASSWORD=no (default).

Here's my compose file:

services:
  redis-base: &redis-base
    image: docker.io/bitnami/redis-cluster:latest
    environment:
      #ALLOW_EMPTY_PASSWORD: yes
      REDIS_MASTER_PASSWORD: myredispassword
      REDIS_PASSWORD: myredispassword
      REDIS_PASSWORD_FILE: /opt/redis/password.txt
      REDIS_NODES: redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5
    volumes:
      - ./pass:/opt/redis

  redis-node-0:
    <<: *redis-base
    container_name: redis-node-0

  redis-node-1:
    <<: *redis-base
    container_name: redis-node-1

  redis-node-2:
    <<: *redis-base
    container_name: redis-node-2

  redis-node-3:
    <<: *redis-base
    container_name: redis-node-3

  redis-node-4:
    <<: *redis-base
    container_name: redis-node-4

  redis-node-5:
    <<: *redis-base
    container_name: redis-node-5
    depends_on:
      - redis-node-0
      - redis-node-1
      - redis-node-2
      - redis-node-3
      - redis-node-4
    environment:
      #ALLOW_EMPTY_PASSWORD: yes
      REDIS_MASTER_PASSWORD: myredispassword
      REDIS_PASSWORD: myredispassword
      REDIS_PASSWORD_FILE: /opt/redis/password.txt
      REDIS_NODES: redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5
      REDIS_CLUSTER_REPLICAS: 1
      REDIS_CLUSTER_CREATOR: yes

redis-node-5 logs shows

Node redis-node-0 not ready, waiting for all the nodes to be ready...

And all nodes are role:master, nobody is a slave.

I have no issue when changing the env variables as below:

      ALLOW_EMPTY_PASSWORD: yes
      #REDIS_MASTER_PASSWORD: myredispassword
      #REDIS_PASSWORD: myredispassword
      #REDIS_PASSWORD_FILE: /opt/redis/password.txt

What am i missing? Any input is appreciated!

Best, Dave