Redis gets stuck in mget

[chaurasiya-P]

I have spring reactive project and I am using RedisReactiveCommand's mget to fetch multiple keys values together. I was iterating over number of keys to fetch data from redis using get method. Later on, I move to mget to fetch data altogether for all keys in single time. But after doing mget changes, redis performance degraded. Response time of service has increased.

I moved to mget to decrease number of IO calls but it seems like redis gets stuck in mget which causes increase in response time. I call mget for number of keys, if data found in redis collect it in list else call backend and save data in redis and collect that too in list and return final list as response. My code is like this:

redis.mget(keys).flatMap(keyvalue -> 
                  if(value is null) call to Backend and return 
                  else return value
).collectList();

[chaurasiya-P]

We are using redis without enabling cluster mode, so mget flow goes through AbstractRedisReactiveCommands class mget. @mp911de

[mp911de]

Can you provide a minimal reproducer that can demonstrate the issue (e.g. a GitHub project including a description of your cluster setup)?

[chaurasiya-P]

@Bean
RedisReactiveCommands<byte[],byte[]> redis() {
    RedisClient client = RedisClient.create(RedisURI.create(host,port));
    StatefulRedisConnection<byte[],byte[]> connection = client.connect(ByteArrayCodec.INSTANCE);
    return connection.reactive();
}

This is how we have created bean of redis. Injected this bean to query to redis. We serialize object into byte[] before saving into redis. Then while reading from redis we deserialize that data. So our code flow for mget goes like this:

redis.mget(redisKeys).flatMap(keyValue -> {  //redisKeys are array of byte[]
    byte[] val = null;
    try {
        val = keyvalue.getValue();
    } catch(Exception e) {
        //Do nothing, just to avoid exception in case of key not found in redis and value is null
    }
    if(null==value) {  //key is not in redis
        call to backend for that key and save in redis as byte[] and return response from backend
    } else {  //value found in redis
        desrialize value and return
    }
}).collectList();

@mp911de check this. It was working fine with redis get for single key, also when in mget, we send only single key it works properly but on increasing number of keys to redis mget, it gets stuck.

[mp911de]

    call to backend for that key and save in redis as byte[] and return response from backend

I suggest, that you invoke subsequent Redis operations outside of flatMap and after collectList. It could well be that the backpressure mechanism awaits completion of the nested Redis calls.

[chaurasiya-P]

@mp911de I tried this too, but this too is getting stuck in redis. Getting intermittent very high response time from redis. Code sample:

return redis.mget(redisKeys).collectList().flatMap(keyValueList -> {
    return Flux.fromIterable(keyValueList).parallel().flatMap(keyValue -> {
        //operation on key value based on value is null or not
    }).sequential().collectList();
});

[chaurasiya-P]

@mp911de check last comment. It too is giving high response time. Can you suggest where It might go wrong.

[chaurasiya-P]

@mp911de Please reply

[tishun]

@mp911de I tried this too, but this too is getting stuck in redis. Getting intermittent very high response time from redis. Code sample:

return redis.mget(redisKeys).collectList().flatMap(keyValueList -> {
    return Flux.fromIterable(keyValueList).parallel().flatMap(keyValue -> {
        //operation on key value based on value is null or not
    }).sequential().collectList();
});

Hey @chaurasiya-P ,

without a good way to reproduce this it would be hard for us to help you.

Have you tried profiling your code to see where the issue happens exactly?