Redis `SCAN` cursor exceeds `Long.MAX_VALUE` resulting in `NumberFormatException`

[jan-domozilov]

Bug Report

Current Behavior

I am using AWS ElastiCache (Redis), the brand new serverless version of it (https://aws.amazon.com/blogs/aws/amazon-elasticache-serverless-for-redis-and-memcached-now-generally-available/). This is the first time I am using AWS ElastiCache at all, so I am not sure whether this is new Serverless ElasticCache specific or maybe ElastiCache related issue in general.

So, I connect to Redis using Lettuce:

public LettuceConnectionFactory redisConnectionFactory() {

        LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
                .readFrom(ReadFrom.REPLICA_PREFERRED)
                .commandTimeout(Duration.ofSeconds(3L))
                .useSsl()
                .build();

        RedisStaticMasterReplicaConfiguration serverConfig = new RedisStaticMasterReplicaConfiguration(host, redisPort);

        return new LettuceConnectionFactory(serverConfig, clientConfig);
}

and I create RedisTemplate like this

public RedisTemplate<String, Object> redisTemplate() {
        final RedisTemplate<String, Object> template = new RedisTemplate<String, Object>();
        template.setConnectionFactory(redisConnectionFactory());
        template.setValueSerializer(new GenericToStringSerializer<Object>(Object.class));
        return template;
}

All good. I can write and read from Redis.

Then I try to do simple scan like that

Cursor c = redisTemplate.scan(ScanOptions.scanOptions().match(cacheName + "*").build());

And I get an exception

java.lang.NumberFormatException: For input string: "9286422431637962772"
    at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[na:na]
    at java.base/java.lang.Long.parseLong(Long.java:708) ~[na:na]
    at java.base/java.lang.Long.parseLong(Long.java:831) ~[na:na]
    at org.springframework.data.redis.connection.lettuce.LettuceScanCursor$LettuceScanIteration.<init>(LettuceScanCursor.java:104) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands$1.doScan(LettuceKeyCommands.java:160) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.connection.lettuce.LettuceScanCursor.scanAndProcessState(LettuceScanCursor.java:73) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.connection.lettuce.LettuceScanCursor.doScan(LettuceScanCursor.java:52) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.core.ScanCursor.scan(ScanCursor.java:90) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.core.ScanCursor.doOpen(ScanCursor.java:132) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.core.ScanCursor.open(ScanCursor.java:121) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands.doScan(LettuceKeyCommands.java:168) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands.scan(LettuceKeyCommands.java:137) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.connection.DefaultedRedisConnection.scan(DefaultedRedisConnection.java:135) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.core.RedisTemplate.lambda$scan$11(RedisTemplate.java:648) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.core.RedisTemplate.executeWithStickyConnection(RedisTemplate.java:523) ~[spring-data-redis-3.1.2.jar!/:3.1.2]
    at org.springframework.data.redis.core.RedisTemplate.scan(RedisTemplate.java:647) ~[spring-data-redis-3.1.2.jar!/:3.1.2]

To get confirmation I connect to Redis via redis-cli and

dns-of-aws-redis:6379> scan 0 match cacheName*
1) "9286422431637962824"
2) 1) key1
     2) key2

Well, 9286422431637962824 returned by AWS Redis as the cursor is bigger than Long.MAX_VALUE and this is the source of the problem.

There is ScanIteration which expects cursorId to be long

public ScanIteration(long cursorId, @Nullable Collection<T> items) {
        this.cursorId = cursorId;
        this.items = (Collection)(items != null ? new ArrayList(items) : Collections.emptyList());
}

and LettuceScanIteration extends ScanIteration

static class LettuceScanIteration<T> extends ScanIteration<T> {
        private final io.lettuce.core.ScanCursor cursor;

        LettuceScanIteration(io.lettuce.core.ScanCursor cursor, Collection<T> items) {
            **super(Long.parseLong(cursor.getCursor()), items);**
            this.cursor = cursor;
        }
}

while io.lettuce.core.ScanCursor treats cursor as String.

Expected behavior/code

Scan should work and not end up in NumberFormatException.

Environment

• spring-data-redis 3.1.2
• Redis version:

INFO command results 

# Server
redis_version:7.1
redis_mode:cluster
arch_bits:64
run_id:0

# Replication
role:master
connected_slaves:1
slave0:ip=somevalue.cache.amazonaws.com,port=6380,state=online,offset=0,lag=0

# Cluster
cluster_enabled:1

Summary

Seems like there are 3 options on a table:

1) It is me who is doing something wrong or I am missing something. Am I doing something wrong? 2) It is specifically AWS Redis doing something wrong returning cursor value to be bigger than Long.MAX_VALUE. I googled trying to find some Redis specification stating rules about cursor value - I was not able to find anything. If such exists could you please refer me to it and then I will report to AWS or Redis depending on what specifications state? 3) If there is no such specification (I still expect there should be one I just was not able to find it) then spring-data-redis states cursor should be Long while there is no strict specification for such

As I was not able to find specification about cursor value by Redis and due to the fact that in the end it is spring-data-redis codebase which does Long.parseLong(cursorValueFromRedis) line of code I decided to start by reporting bug first here.

Quick workaround

Could you please suggest a quick workaround if possible?

[mp911de]

Redis uses an unsigned 64 bit long value whereas our implementation uses the signed variant. We need to fix this bug.

[wleo04]

@mp911de I'm having a similar problem right now, and if the issue is before the revision, can I fix it and post PR?

[mp911de]

Unfortuneately, this change isn't straightforward as the cursor is being used across the ScanCursor and ScanIteration types including method signatures. We can make it work, but it will require a wider scope of changes.