RedisLabs / redis-cluster-proxy

A proxy for Redis clusters.
GNU Affero General Public License v3.0
993 stars 132 forks source link

Redis-proxy startup fails when a redis cluster node is down #110

Open hac3ru opened 1 year ago

hac3ru commented 1 year ago

Hello,

First, let me start by saying that I've been waiting for this product for quite some time, as it makes all our lives easier! I've got the following setup: Running 6 redis instances containers (for simplicity), using docker-compose with network_mode: host (for simplicity). I've started redis-cluster-proxy using command: -p 6379 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006

Everything is well, until one of the nodes are down and I try to start up redis-cluster-proxy. How to reproduce: Build a container with redis-cluster-proxy from github, branch 1.0, using the Dockerfile provided by bsergean on Issue #23 (https://github.com/RedisLabs/redis-cluster-proxy/issues/23) by running docker build -t redis-proxy .

Use the following docker-compose.yml file:


services:
  proxy:
    image: redis-proxy
    container_name: proxy
    network_mode: host
    command: -p 6379 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006

  r1:
    image: redis:7.0.2-alpine
    container_name: r1
    volumes:
      - ./redis.conf:/etc/redis/redis.conf
    network_mode: host
    command: /etc/redis/redis.conf --port 7001

  r2:
    image: redis:7.0.2-alpine
    container_name: r2
    volumes:
      - ./redis.conf:/etc/redis/redis.conf
    network_mode: host
    command: /etc/redis/redis.conf --port 7002

  r3:
    image: redis:7.0.2-alpine
    container_name: r3
    volumes:
      - ./redis.conf:/etc/redis/redis.conf
    network_mode: host
    command: /etc/redis/redis.conf --port 7003

  r4:
    image: redis:7.0.2-alpine
    container_name: r4
    volumes:
      - ./redis.conf:/etc/redis/redis.conf
    network_mode: host
    command: /etc/redis/redis.conf --port 7004

  r5:
    image: redis:7.0.2-alpine
    container_name: r5
    volumes:
      - ./redis.conf:/etc/redis/redis.conf
    network_mode: host
    command: /etc/redis/redis.conf --port 7005

  r6:
    image: redis:7.0.2-alpine
    container_name: r6
    volumes:
      - ./redis.conf:/etc/redis/redis.conf
    network_mode: host
    command: /etc/redis/redis.conf --port 7006

Create the actual cluster using redis-cli --port 7001 --cluster create --cluster-replicas 1 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006

Start the proxy using docker restart proxy and inspect the logs docker logs proxy. Everything should be fine. Stop one instance: docker stop r1

Restart the proxy: docker restart proxy and inspect the logs docker logs proxy.

This error shows up:

[2022-10-07 06:27:55.919/M] Redis Cluster Proxy v0.9.102
[2022-10-07 06:27:55.919/M] Commit: (00000000/0)
[2022-10-07 06:27:55.919/M] PID: 1
[2022-10-07 06:27:55.919/M] OS: Linux 5.4.0-126-generic x86_64
[2022-10-07 06:27:55.919/M] Bits: 64
[2022-10-07 06:27:55.919/M] Log level: info
[2022-10-07 06:27:55.919/M] Connections pool size: 10 (respawn 2 every 50ms if below 10)
[2022-10-07 06:27:55.920/M] Listening on *:6379
[2022-10-07 06:27:55.920/M] Starting 8 threads...
[2022-10-07 06:27:55.920/M] Fetching cluster configuration...
[2022-10-07 06:27:55.920/M] Could not connect to Redis at 127.0.0.1:7001: Connection refused
Could not connect to Redis at 127.0.0.1:7001: Connection refused
[2022-10-07 06:27:55.921/M] ERROR: Failed to fetch cluster configuration!
[2022-10-07 06:27:55.922/M] FATAL: failed to create thread 0.
codifierr commented 1 year ago

Any update on this? We are also facing the same.

thanghp2711 commented 1 year ago

I'm facing this issue too, any solutions?