search

RedisLabs / redis-cluster-proxy

A proxy for Redis clusters.
GNU Affero General Public License v3.0
990 stars 129 forks source link

When startup and secondary nodes are down, proxy can not start #36

Open t4410 opened 4 years ago

t4410 commented 4 years ago

When startup and secondary nodes are down(not first node), redis-cluster-proxy can not start. Is there any way to start the proxy?

[2020-02-26 00:39:15.620/M] Redis Cluster Proxy v999.999.999 (unstable)
[2020-02-26 00:39:15.620/M] Commit: (4490fec3/0)
[2020-02-26 00:39:15.620/M] Git Branch: unstable
[2020-02-26 00:39:15.620/M] Cluster Address: redis-0-0:50080
[2020-02-26 00:39:15.620/M] PID: 1
[2020-02-26 00:39:15.620/M] OS: Linux 3.10.0-862.14.4.el7.x86_64 x86_64
[2020-02-26 00:39:15.620/M] Bits: 64
[2020-02-26 00:39:15.620/M] Log level: info
[2020-02-26 00:39:15.620/M] Listening on *:6379
[2020-02-26 00:39:15.620/M] Starting 8 threads...
[2020-02-26 00:39:15.621/M] Fetching cluster configuration...
Could not connect to Redis at 10.42.1.142:50081: No route to host
[2020-02-26 00:39:17.883/M] ERROR: Failed to fetch cluster configuration!
[2020-02-26 00:39:17.883/M] FATAL: failed to create thread 0.

cluster.c/fetchClusterConfiguration

while ((ln = listNext(&li))) {
    clusterNode *friend = ln->value;
    success = clusterNodeLoadInfo(cluster, friend, NULL, NULL); // I guess error has occurred here
    if (!success) {
        listDelNode(friends, ln);
        freeClusterNode(friend);
        goto cleanup;
    }
    clusterAddNode(cluster, friend);
}
artix75 commented 4 years ago

The Proxy currently requires that all nodes of the cluster must be up at startup when it fetches the cluster's internal map. I'll probably change this in the next weeks.

wangyi24420 commented 4 years ago

the same issue occur when one of master node down,and redis cluster failover successful,but redis-clutser-proxy cannot startup.

AllenDou commented 4 years ago

[2020-05-09 15:46:02.566/M] Redis Cluster Proxy v999.999.999 (unstable) [2020-05-09 15:46:02.566/M] Commit: (ac83840d/1) [2020-05-09 15:46:02.566/M] Git Branch: unstable [2020-05-09 15:46:02.566/M] PID: 18180 [2020-05-09 15:46:02.566/M] OS: Linux 4.16.3-301.fc28.x86_64 x86_64 [2020-05-09 15:46:02.566/M] Bits: 64 [2020-05-09 15:46:02.566/M] Log level: info [2020-05-09 15:46:02.566/M] Connections pool size: 10 (respawn 2 every 50ms if below 10) [2020-05-09 15:46:02.567/M] The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. [2020-05-09 15:46:02.567/M] Listening on *:7777 [2020-05-09 15:46:02.567/M] Starting 8 threads... [2020-05-09 15:46:02.567/M] Fetching cluster configuration... Could not connect to Redis at 127.0.0.1:6379: Connection refused [2020-05-09 15:46:02.568/M] ERROR: Failed to fetch cluster configuration! [2020-05-09 15:46:02.569/M] FATAL: failed to create thread 0.

same issue, when master node(6379) is down, slave become the master node, redis-cluster-proxy hangs, and even can not restart.

Ops-learning commented 3 years ago

Has this problem been solved yet? The same thing happened to me during my tests

Below is my Proxy log [2020-09-09 11:23:21.305/M] Redis Cluster Proxy v0.9.102 [2020-09-09 11:23:21.305/M] Commit: (00000000/0) [2020-09-09 11:23:21.305/M] PID: 3441 [2020-09-09 11:23:21.305/M] OS: [2020-09-09 11:23:21.305/M] Bits: 64 [2020-09-09 11:23:21.305/M] Log level: debug [2020-09-09 11:23:21.305/M] Connections pool size: 50 (respawn 1 every 2ms if below 45) [2020-09-09 11:23:21.305/M] Listening on :22121 [2020-09-09 11:23:21.305/M] Pidfile: '/var/run/redis-cluster-proxy.pid' [2020-09-09 11:23:21.306/M] Starting 5 threads... [2020-09-09 11:23:21.306/M] Creating thread 0... [2020-09-09 11:23:21.306/M] Fetching cluster configuration... [2020-09-09 11:23:21.306/M] Trying cluster entry point x.x.x.x:6379 [2020-09-09 11:23:21.306/M] Fetching cluster configuration from entry point 'x.x.x.x:6379' [2020-09-09 11:23:21.307/M] ERROR: Failed to fetch cluster configuration! [2020-09-09 11:23:21.307/M] Freeing thread 0 [2020-09-09 11:23:21.307/M] Free shared cluster on thread 0 [2020-09-09 11:23:21.307/M] FATAL: failed to create thread 0. [2020-09-09 11:23:34.324/M] Redis Cluster Proxy v0.9.102 [2020-09-09 11:23:34.324/M] Commit: (00000000/0) [2020-09-09 11:23:34.324/M] PID: 3477 [2020-09-09 11:23:34.324/M] OS: [2020-09-09 11:23:34.324/M] Bits: 64 [2020-09-09 11:23:34.324/M] Log level: debug [2020-09-09 11:23:34.324/M] Connections pool size: 50 (respawn 1 every 2ms if below 45) [2020-09-09 11:23:34.325/M] Listening on :22121 [2020-09-09 11:23:34.325/M] Pidfile: '/var/run/redis-cluster-proxy.pid' [2020-09-09 11:23:34.325/M] Starting 5 threads... [2020-09-09 11:23:34.325/M] Creating thread 0... [2020-09-09 11:23:34.326/M] Fetching cluster configuration... [2020-09-09 11:23:34.326/M] Trying cluster entry point x.x.x.x:6379 [2020-09-09 11:23:34.326/M] Fetching cluster configuration from entry point 'x.x.x.x:6379' [2020-09-09 11:23:34.327/M] ERROR: Failed to fetch cluster configuration! [2020-09-09 11:23:34.327/M] Freeing thread 0 [2020-09-09 11:23:34.327/M] Free shared cluster on thread 0 [2020-09-09 11:23:34.327/M] FATAL: failed to create thread 0.

ghost commented 3 years ago

Any chance to get build, which won't require all nodes up on proxy startup?

[2021-06-07 13:46:46.900/M] Redis Cluster Proxy v999.999.999 (unstable) [2021-06-07 13:46:46.900/M] Commit: (ac83840d/0) [2021-06-07 13:46:46.900/M] Git Branch: unstable [2021-06-07 13:46:46.900/M] PID: 257911 [2021-06-07 13:46:46.901/M] OS: Linux 2.6.32-754.35.1.el6.x86_64 x86_64 [2021-06-07 13:46:46.901/M] Bits: 64 [2021-06-07 13:46:46.901/M] Log level: debug [2021-06-07 13:46:46.901/M] Connections pool size: 10 (respawn 2 every 50ms if below 10) [2021-06-07 13:46:46.901/M] Listening on 127.0.0.1:7777 [2021-06-07 13:46:48.120/M] Pidfile: '/var/run/redis-cluster-proxy.pid' [2021-06-07 13:46:48.121/M] Starting 8 threads... [2021-06-07 13:46:48.121/M] Creating thread 0... [2021-06-07 13:46:48.121/M] Fetching cluster configuration... [2021-06-07 13:46:48.121/M] Trying cluster entry point x.x.x.x:6379 [2021-06-07 13:46:48.121/M] Fetching cluster configuration from entry point 'x.x.x.x:6379' [2021-06-07 13:46:48.124/M] ERROR: Failed to fetch cluster configuration! [2021-06-07 13:46:48.124/M] Freeing thread 0 [2021-06-07 13:46:48.125/M] Free shared cluster on thread 0 [2021-06-07 13:46:48.125/M] FATAL: failed to create thread 0.

pradeep13143 commented 1 year ago

Hi Team, any update on this ticket?