Improve polling for Redis Streams [DATAREDIS-964]

Akash Kengunte Jayachandra opened DATAREDIS-964 and commented

After discussion with Mark Paluch in https://gitter.im/spring-projects/spring-data

We at Intuit want to use spring-data-redis for redis-streams and as discussed, when I did a POC with reactive version of spring-data-redis, I noticed that there is a new connection open for every stream that is subscribed. This would not be a scalable solution for us as we will be subscribing to large number of streams at a time.

As we discussed it would be helpful if we can use connections from connection pool and do a XREAD in a round robin so that we can limit number of connections that are established and use the connections form the connection pool.

Could we please discuss more on this?

Usecase:

Given a node there will be 10,000 streams that we are listening/subscribing to at a time. That means using spring-data-redis we will have 10,000 connections open with redis streams. We do not want that and would like to use connection pooling, that way if we limit the number to 500 available in the connection pool then we would be subscribing to 10,000 streams using that 500 connections.

I tried this out and set the connection pool to spring.redis.lettuce.pool.max-active=9, Runtime.getRuntime().availableProcessors() = 8 and number of streams we are subscribing to are 12. In this case we are getting an error saying

2019-04-05 13:31:49.383 ERROR 22808 --- [ioEventLoop-4-8] reactor.core.publisher.Operators         : Operator called default onErrorDropped

reactor.core.Exceptions$ErrorCallbackNotImplemented: java.util.concurrent.CompletionException: java.util.NoSuchElementException: Pool exhausted

So, using spring-data-redis subscribe to number of streams that is more than number of connections available in the connection pool? will spring-data-redis handle that?

Affects: 2.2 M2 (Moore)

spring-projects / spring-data-redis

Improve polling for Redis Streams [DATAREDIS-964] #1538