redis / lettuce

Advanced Java Redis client for thread-safe sync, async, and reactive usage. Supports Cluster, Sentinel, Pipelining, and codecs.
https://lettuce.io
MIT License
5.3k stars 949 forks source link

Getting the keys from cache returns the wrong amount in cluster mode with replica enabled. #2839

Open andrew-afs opened 2 months ago

andrew-afs commented 2 months ago

Bug Report

Current Behavior

I am experiencing an issue with getting the number of keys in the cache. I am setting the keys with connection.async.set(key, value). I also have a method to get the number of keys: connection.sync.keys("*").size(). The issue is when I run the unit tests, after I insert one key value pair and use the size method, it returns 2. When I print the keys themselves I get something like this list("123", "123"). I do have replica enabled with redis cluster so I setup the connection to only read from master: connection.setReadFrom(ReadFrom.MASTER) however, the issue still persists. When I turn off replica mode in redis, the issue goes away.

I have lettuce setup to use connection pooling for my application. I based the setup off of AWS ElastiCache recommended config https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/BestPractices.Clients-lettuce.html

Input Code

Input Code ```java override def put(key: String, value: T): Future[Unit] = { val compositeKey: String = getCompositeKey(key) val connection = redisClient.acquire.get val result: Future[Unit] = cfgCtx.cacheExpiry match { case Some(v) => connection.async.set(compositeKey, value, new SetArgs().px(v.toMillis)).toScala.map(_ => { if (localCache.isDefined) { logger.debug("successfully added/updated key to redis. storing to local cache") storeToLocalCache(compositeKey, value) } Future.unit }) case None => connection.async.set(compositeKey, value).toScala.map(_ => { if (localCache.isDefined) { logger.debug("successfully added/updated key to redis. storing to local cache") storeToLocalCache(compositeKey, value) } Future.unit }) } result.onComplete(_ => redisClient.release(connection)) result } override def size: Long = { val connection = redisClient.acquire.get connection.setReadFrom(ReadFrom.MASTER) val result = connection.sync.keys(getCompositeKey("*")).size() redisClient.release(connection) result } //Unit tests that always fails "cache entry should be one after input" in { Await.result(redisCacheNoLocalCache.put("abc", "xyz"), Duration.Inf) redisCacheNoLocalCache.size shouldBe 1 } ```

Expected behavior/code

When getting the keys from the cache using ReadFrom.MASTER, it should not return the replica keys.

Environment

Possible Solution

Additional context

tishun commented 2 months ago

Hey @andrew-afs ,

I am not able to replicate this locally, can you give some more details on the deployment you are using? Could you also verify if the same results are returned when using another client, for ex. redis-cli?

If the same results are returned from both clients, then the issue is in the setup / server.