Does redistimeseries-go support RedisTimeSeries Cluster?

[racc-costa]

Hi RedisTimeSeries team,

The RedisTimeSeries module brings many interesting features that can expand the way we use Redis. But looking at the examples and the documentation, if I got it right, redistimeseries-go does not support cluster? If not, is there any plan to add cluster support or any suggestion to use the RedisTimeSeries cluster in prod with Go?

Thank you!

[filipecosta90]

Hi RedisTimeSeries team,

The RedisTimeSeries module brings many interesting features that can expand the way we use Redis. But looking at the examples and the documentation, if I got it right, redistimeseries-go does not support cluster? If not, is there any plan to add cluster support or any suggestion to use the RedisTimeSeries cluster in prod with Go?

Thank you!

Hi there Ricardo, breaking the answer parts:

• redistimeseries-go is based on redigo. we're working on removing the direct bound to redigo ( since redigo does not support OSS cluster )
• in the future the redistimeseries-go client will be as agnostic as possible with regards the underlying client/setup
• In the meantime you can use the vanilla redis-go-cluster or radix ( if you can share one sample querie/use case we can prepare some examples )

Let's keep this issue open an use it to track when we will support OSS cluster within this client.

[racc-costa]

Thanks for your answer Filipe,

To start, I need to add and queries timeseries as follows:

TS.ADD temperature:2:32 1548149180000 26 LABELS sensor_id 2 area_id 32

TS.MRANGE 1548149180000 1548149210000 AGGREGATION avg 5000 FILTER area_id=32 sensor_id!=1

So, basically I would need TS.ADD and TS.MRANGE, do you have any proposal on how it would be to implement this in one of these two libraries?

[franklinlindemberg]

Hey @filipecosta90 tks a lot for your suggestion. We managed to implement the commands we needed (TS.ADD and TS.MRANGE) using the suggestion of radix library (redis-go-cluster didnt work, I think it was something related to the cluster connection, it connected but when the command was executed there was a nil error. Didnt dig deep into the problem since it worked with radix).

Here are both commands using radix in case anyone else need to use.

TS.ADD (Serialize is the method from here: https://github.com/RedisTimeSeries/redistimeseries-go/blob/2169513f3970dcb84439fbdd0dd9ed218ccbb1ec/common.go#L87

        args := []string{key, strconv.FormatInt(timestamp, 10), floatToStr(value)}
    args, err = options.Serialize(args)
    if err != nil {
        return
    }
    c.Client.Do(radix.Cmd(nil, "TS.ADD", args...))

TS.MRANGE:

        args := []string{}
    args = append(args, strconv.FormatInt(fromTimestamp, 10))
    args = append(args, strconv.FormatInt(toTimestamp, 10))
    args = append(args, strconv.FormatInt(mrangeOptions.Count, 10))
    args = append(args, "AGGREGATION", mrangeOptions.AggregationType, strconv.Itoa(mrangeOptions.TimeBucket))
    if mrangeOptions.WithLabels == true {
        args = append(args, "WITHLABELS")
    }
    args = append(args, "FILTER")
    for _, filter := range filters {
        args = append(args, filter)
    }

    var reply interface{}
    c.Client.Do(radix.Cmd(&reply, "TS.MRANGE", args...))

[jmesquita]

As it turns out, no, this didn't work. I'm sure @franklinlindemberg can give a lot more details but radix fails to find the right instance of redis that has the data when using MGET and MRANGE, even if we guarantee that all the keys required to make the computation would be on the same instance. I guess there's no way that the protocol can tell which slot and therefore which service it will be on without the key itself. @filipecosta90 are we on the right track here? If not using cluster, I'm assuming most users are probably using sentinel? Guidance would be much appreciated.

[filipecosta90]

As it turns out, no, this didn't work. I'm sure @franklinlindemberg can give a lot more details but radix fails to find the right instance of redis that has the data when using MGET and MRANGE, even if we guarantee that all the keys required to make the computation would be on the same instance. I guess there's no way that the protocol can tell which slot and therefore which service it will be on without the key itself. @filipecosta90 are we on the right track here? If not using cluster, I'm assuming most users are probably using sentinel? Guidance would be much appreciated.

Will prepare a quick example @jmesquita and @franklinlindemberg . Should have it until EOD :)

[jmesquita]

Oh this is so kind! Obrigado!

[filipecosta90]

hi there @jmesquita and @franklinlindemberg the following gist should exemplify the creation, ingestion and querying of multiple time series using radix: https://gist.github.com/filipecosta90/4325150c346e31365938d863c11d7fd0

a quick example using a 2 node cluster:

$ ./radix-redistimeseries-example --host localhost:20000 --cluster-mode
[1 1]
[2 2]
[3 3]
[4 4]
[5 5]
[6 6]
[7 7]
[8 8]
[9 9]
[10 10]
temperature:{area_32}:1
map[area_id:32 sensor_id:1]
map[1:2 2:4]
temperature:{area_32}:2
map[area_id:32 sensor_id:2]
map[1:2 2:4]

to use this same example on standalone redis instances:

$ ./radix-redistimeseries-example --host localhost:6379 
[1 1]
[2 2]
[3 3]
[4 4]
[5 5]
[6 6]
[7 7]
[8 8]
[9 9]
[10 10]
temperature:{area_32}:1
map[area_id:32 sensor_id:1]
map[1:2 2:4]
temperature:{area_32}:2
map[area_id:32 sensor_id:2]
map[1:2 2:4]

please do provide feedback on whether this example covers what you guys wanted or if you need further examples, etc... :)

[franklinlindemberg]

@filipecosta90 thanks for sharing the code!

I've redid the tests using the same cluster instantiation that you did and it didn't work. Im not sure if you tested the same way (since when you instantiate the cluster you only pass one host, then every key will fall into the same host and TS.MRANGE/TS.MGET will work).

I'll explain how we are using: We have a redistimeseries cluster with 9 nodes (3 masters).

The scenario I tested, the TS.ADD added a key in the second master. When we perform the TS.MGET it always returns empty. I'm pretty sure it's because it's always trying to run the MGET in the first or third master, but never in the second.

From what I read, the way the cluster knows in which instance to look the data for is by hashing the key using {} what I did when running the TS.ADD. But since TS.MGET doesn't have a key, so the cluster can discover the right instance, it gets one the first one.

Am I missing something here?

[filipecosta90]

hi there @franklinlindemberg breaking the answer by parts:

since when you instantiate the cluster you only pass one host, then every key will fall into the same host and TS.MRANGE/TS.MGET will work

when you call radix.NewCluster it will underneeth call Sync which basically synchronize the Cluster with the actual redis cluster, making new pools to all instances ( even tough I only passed an initial single point of contact ). I've updated the gist to print the topology. Please check the following link: https://gist.github.com/filipecosta90/4325150c346e31365938d863c11d7fd0#file-radix-example-for-redistimeseries-oss-cluster-connection-L69

Regarding MGET and MRANGE you're totally right. To solve it we need to iterate over the topology and isue the command to each participating node ( master's ). The gist has also been updated to it here: https://gist.github.com/filipecosta90/4325150c346e31365938d863c11d7fd0#file-radix-example-for-redistimeseries-oss-cluster-connection-L111

Regarding the expected output, for a 3 node cluster:

$ ./radix-redistimeseries-example --host localhost:20000 --cluster-mode
Cluster topology: [{127.0.0.1:20000 e60215b65b249e3f292bcd9635a56a3dc21b5ab0 [[%!s(uint16=0) %!s(uint16=5462)]]  } {127.0.0.1:20002 5b3e35f20dd36d9aa71ee92e5d6dc15e1d47f7cd [[%!s(uint16=5462) %!s(uint16=10924)]]  } {127.0.0.1:20004 33b76b65bb50934637d415b6b5af1a0a7e072d20 [[%!s(uint16=10924) %!s(uint16=16384)]]  }]
[1 1]
[2 2]
[3 3]
[4 4]
[5 5]
[6 6]
[7 7]
[8 8]
[9 9]
[10 10]
key1
map[area_id:32]
map[1:10]
temperature:{area_32}:1
map[area_id:32 sensor_id:1]
map[1:2 2:4]
temperature:{area_32}:2
map[area_id:32 sensor_id:2]
map[1:2 2:4]

[franklinlindemberg]

@filipecosta90 Awesome! it works. Indeed we were thinking about how to loop the nodes to perform the action and do a merge using each response.

What Im doing now to improve the performance is do some filtering logic to not reach the 9 nodes of the cluster (we just need to hit one from master/replica). I saw that from the topology I'm able to know if its a replica or slave, so I could reduce the requests to 3.

I have one more question, that I couldn't find an answer yet. We are hashing the key in a subset of the key key1.{key2}.key3, so if we have the same key2, TS.ADD will always store that data in the same slot. Let's say when I'm performing the TS.MRANG, i have the key2 value. Is it possible to discover which slot that key is mapped to? If this is possible I could reduce the amount of requests to only one instead of 3

[franklinlindemberg]

@filipecosta90 I found a method in radix that given a queue we can find the respective slot number

func ClusterSlot(key []byte) uint16

Having this in hand its possible to find the right instance and do only one request

[filipecosta90]

@filipecosta90 I found a method in radix that given a queue we can find the respective slot number

func ClusterSlot(key []byte) uint16

Having this in hand its possible to find the right instance and do only one request

Hi there @franklinlindemberg please take into consideration that it will only retrieve you the keys living in the slots for the cluster node you're targeting. MGET and MRANGE are based uppon a condition ( and that conditions is checked for the keys living in the node ) so you will always have to connect to each of the nodes and aggregate on the client.

[franklinlindemberg]

@filipecosta90 Tks for you consideration and support!

We managed to run it with cluster! for our use case we guarantee that all data we want to mget/mrange is only in the same slot (using {} on ts.add) so we can reach only a instance that handles that slot on mget/mrange.

[uptycs-anudeep]

Any updates on redistimeseries-go becoming agnostic to underlying cluster/single node redis ?

[filipecosta90]

Any updates on redistimeseries-go becoming agnostic to underlying cluster/single node redis ?

Hi there @uptycs-anudeep this is something we've been discussing internally on how to properly support it, without breaking the current client, and still being as performant as possible. We're thinking about different approaches to it but in the meantime community help is deeply accepted :)

We would like to have you guys opinion, POCs, etc... to help up decide better on how should we tackle this. Adding the help-wanted tag and lets give some time to the community to pronounce themselves on this issue.

[mcauto]

It helped me a lot :) Are there any additional updates or ongoing operations for the cluster?