redis / go-redis

Redis Go client
https://redis.uptrace.dev
BSD 2-Clause "Simplified" License
20.2k stars 2.38k forks source link

Redis Clustered Pubsub Delayed SSubscribe triggers MOVE error. #3099

Open CR7273868 opened 2 months ago

CR7273868 commented 2 months ago

Redis or Redis GO does not go well with Clustered PUBSUB of Redis

Expected Behavior

Redis GO works even when using the same PUBSUB pointer to subscribe to other channels later on.

Current Behavior

Redis GO does SSubscribe, but then if u do one later one let's say in a goroutine. It says MOVED to another IP or PORT. And doesn't receive the messages at all, afterwards.

Possible Solution

Keep track of shards where what comes from. Fred from Redis RS provides a way to see what shard channel a server was bound to. Per message u'd receive, you could see which server it was hooked too, if I am not wrong.

FRED RS

Steps to Reproduce

package main

import (
    "context"
    "log"
    "time"

    "github.com/redis/go-redis/v9"
)

func main() {
    client := redis.NewClusterClient(&redis.ClusterOptions{
        Addrs: []string{"masked:7000"},
    })

    client_2 := redis.NewClusterClient(&redis.ClusterOptions{
        Addrs: []string{"masked:7000"},
    })

    pubsub := client.SSubscribe(context.Background(), "mychannel")
    defer pubsub.Close()

    go func() {
        // Subscribe to a channel
        time.Sleep(2 * time.Second)
        if err := pubsub.SSubscribe(context.Background(), "mychannel2"); err != nil {
            panic(err)
        }
        log.Println("Subscribed to mychannel2")
    }()
    go func() {
        ticker := time.NewTicker(1 * time.Second)
        for range ticker.C {
            // Channel 2 does not work when ssubscribing later
            client_2.SPublish(context.Background(), "mychannel2", "hello")
            // Does work since we subscribed when we started the program
            client.SPublish(context.Background(), "mychannel", "hello")
        }
    }()
    for {
        msg, err := pubsub.ReceiveMessage(context.Background())
        if err != nil {
            log.Println("ERROR", err)
            continue
        }
        log.Println(msg.Channel, msg.Payload)
    }

}

Output log:

2024/09/01 07:31:00 mychannel hello
2024/09/01 07:31:01 Subscribed to mychannel2
2024/09/01 07:31:01 ERROR MOVED 4312 masked:7002
2024/09/01 07:31:01 mychannel hello
2024/09/01 07:31:02 mychannel hello
2024/09/01 07:31:03 mychannel hello
2024/09/01 07:31:04 mychannel hello
2024/09/01 07:31:05 mychannel hello
2024/09/01 07:31:06 mychannel hello

All you need is a Redis Cluster and use SSUBSCRIBE and have it move across redis nodes to replicate this issue. I have tried several cluster setups correctly, but keep running into this issue. I assume IOREDIS from Node.JS has this exact same issue. The only that works is Fred from Redis which I actively use.

Context (Environment)

4 servers: Each has 3 master coupled with slave to each master. Granting 12 masters coupled with 12 slaves. Each server is compiled with Redis:

Redis server v=7.2.5 sha=00000000:0 malloc=jemalloc-5.3.0 bits=64 build=b5c2f37750637637

Making Redis PUBSUB work correctly, without having to deal with weird issues like moving and it goes out of scope

Redis Clustered PUBSUB doesn't work after subscribing later on.