Performance issues

[amerpersonal]

Hi,

We're using scala-redis client to get things from Redis inside web server. Here is an architecture in brief:

1. Server is implemented pon Akka HTTP
2. On each request, we're getting list of terminal symbols from Redis, using Redis Poll.

This lasts for 3-4 seconds on our dev env. I tried getting same key directly from redis console and its really fast. I isolated the code and here it is (accessing remote Redis server, not localhost):

val redisPoll = new RedisClientPool("cache", 6379)

def getRedisSymbolsAsync(terminalId: Long) = {
      Future {
        redisPoll.withClient { client =>
          val data = client.get(s"key_prefix_${terminalId}")

          data
        }
      }
    }

Then I created a loop to test it

for ( iteration <- 0 to 4) {
      for (i <- 0 until keys.size) {
        s = System.currentTimeMillis()
        getRedisKeyAsync().onComplete {
          case Success(r) => logger.info(s"got redis symbols async in ${System.currentTimeMillis() - s} ms: ${r}")
          case Failure(ex) => logger.error("Failure on getting Redis symbols")
        }
      }
    }

So there are 60 iterations. Here is a part of log

got redis symbols async in 4857 ms got redis symbols async in 4903 ms got redis symbols async in 4968 ms

Key value is very short string.

Are we doing it wrong to open new client from poll on each request or what's the issue? Can you please advise?

Thanks,

Amer

[debasishg]

I think opening a new client on every poll is expensive. Can u allocate a pool of clients for once and then use the pool to fetch a client for every request ?

[amerpersonal]

Thanks for answer @debasishg . Actually, I'm already doing that. I only create one client poll on server start and then call withClient method on each request.

Newest benchmark says it needs 726 ms on my local machine. However, on DEV environment it takes much more, around 3-4 seconds sometimes. Do you have ideas how its possible? Of course, dev env is more used than my local machine.

BTW, I also tried node.js client and its extremely fast, it takes only 30 ms.

Go lang client takes around 600 ms on my local machine.

If you can recommend something to optimise it and have a same response time regardless of requests frequency, it would be great!

[debasishg]

Can u please check what timings you get in the benchmark by making the call synchronous and removing the Future {} ? Besides the normal overhead of closures that get created in the process, I don't see any place for further optimization here.

Also if you have more requests in DEV you can try increasing the maxIdle argument in the call to RedisClientPool - currently it is set to 8 (https://github.com/debasishg/scala-redis/blob/master/src/main/scala/com/redis/Pool.scala#L39).

Let me know how this goes.

[amerpersonal]

Thanks for the answer. Just tried without using futures and heres a result:

got one valye async in 421 ms got one value sync in 308 ms

I also tried changing max idle value (both lower and higher than 8) but it does not affects overall time so much, at least locally.

It's a bit faster now overall.

I'll try to put it in the actor and use the sync version so we'll see how it behaves on DEV env. Will let you know...

[amerpersonal]

Unfortunatelly, I didn't manager to get it faster on DEV. I tried another Scala async client, and it works much faster. Funny thing is that event on PROD it works ok (with much higher load). It can be a Redis internal thing, or the way Futures are handled internally. Thanks for help anyway!