debasishg / scala-redis

A scala library for connecting to a redis server, or a cluster of redis nodes using consistent hashing on the client side.
1.02k stars 219 forks source link

Getting java.lang.Exception: Protocol error: Got (+,[B@38466868) as initial reply byte while pushing to redis quite frequently #271

Open ShreyasTandale opened 3 years ago

ShreyasTandale commented 3 years ago

We are using below redis version setup using bitnami as a master slave configuration. Steps to install - https://engineering.bitnami.com/articles/deploy-and-scale-a-redis-cluster-on-kubernetes-with-bitnami-and-helm.html

Redis version - 5.0.7-debian-10-r32 scala-redis version - 3.30 Frequently, we are getting below exception while pushing keys to redis. java.lang.Exception: Protocol error: Got (+,[B@38466868) as initial reply byte at com.redis.Reply$$anonfun$8.applyOrElse(RedisProtocol.scala:120) at com.redis.Reply$$anonfun$8.applyOrElse(RedisProtocol.scala:118) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) at com.redis.Reply$$anonfun$queuedReplyLong$1.applyOrElse(RedisProtocol.scala:127) at com.redis.Reply$$anonfun$queuedReplyLong$1.applyOrElse(RedisProtocol.scala:127) at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) at com.redis.Reply$$anonfun$2.applyOrElse(RedisProtocol.scala:59) at com.redis.Reply$$anonfun$2.applyOrElse(RedisProtocol.scala:59) at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) at com.redis.Reply$class.receive(RedisProtocol.scala:139) at com.redis.RedisClient.receive(RedisClient.scala:96) at com.redis.R$class.asLong(RedisProtocol.scala:250) at com.redis.RedisClient.asLong(RedisClient.scala:96) at com.redis.ListOperations$$anonfun$lpush$1.apply(ListOperations.scala:10) at com.redis.ListOperations$$anonfun$lpush$1.apply(ListOperations.scala:10) at com.redis.Redis$class.send(RedisClient.scala:31) at com.redis.RedisClient.send(RedisClient.scala:96) at com.redis.ListOperations$class.lpush(ListOperations.scala:10) at com.redis.RedisClient.lpush(RedisClient.scala:96) at com.here.probe.consolidation.redis.RedisCacheService$$anonfun$insert$1$$anonfun$apply$1.apply(RedisCacheService.scala:19) at com.here.probe.consolidation.redis.RedisCacheService$$anonfun$insert$1$$anonfun$apply$1.apply(RedisCacheService.scala:18) at scala.collection.immutable.List.map(List.scala:284) at com.here.probe.consolidation.redis.RedisCacheService$$anonfun$insert$1.apply(RedisCacheService.scala:18) at com.here.probe.consolidation.redis.RedisCacheService$$anonfun$insert$1.apply(RedisCacheService.scala:16) at com.redis.RedisClientPool.withClient(Pool.scala:63) at com.here.probe.consolidation.redis.RedisCacheService$class.insert(RedisCacheService.scala:16) at com.here.probe.consolidation.utils.DependencyInjection$$anon$13.insert(DependencyInjection.scala:63) at com.here.probe.consolidation.services.PublisherService$$anonfun$saveInCache$1$$anonfun$apply$1.apply(PublisherService.scala:30) at com.here.probe.consolidation.services.PublisherService$$anonfun$saveInCache$1$$anonfun$apply$1.apply(PublisherService.scala:26) at scala.collection.immutable.List.map(List.scala:288) at com.here.probe.consolidation.services.PublisherService$$anonfun$saveInCache$1.apply(PublisherService.scala:26) at com.here.probe.consolidation.services.PublisherService$$anonfun$saveInCache$1.apply(PublisherService.scala:24) at scala.collection.immutable.List.map(List.scala:288) at com.here.probe.consolidation.services.PublisherService$class.saveInCache(PublisherService.scala:24) at com.here.probe.consolidation.utils.DependencyInjection$$anon$21.saveInCache(DependencyInjection.scala:118) at com.here.probe.consolidation.services.PublisherService$$anonfun$publishBatchAsync$1.apply(PublisherService.scala:45) at com.here.probe.consolidation.services.PublisherService$$anonfun$publishBatchAsync$1.apply(PublisherService.scala:41) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Code snippet for pushing to redis attached. See class RedisCacheService.scala RedisCacheService RedisConnectionUtil

ShreyasTandale commented 3 years ago

I have observed that our PODS don't process anything for 4-5 HRS, until the next iteration starts. After this idle time, the exception starts coming.

debasishg commented 2 years ago

Hi @ShreyasTandale - Is this issue still there ? Can u please give a reproducible test case for this ?

lenin122 commented 1 year ago

WE have similar issue on ElasstiCache with client 3.42. But don't know how to reproduce it: it comes suddenly and no certain steps can be provided to reproduce it.