Optimal use of Carmine with core.async

[jeroenvandijk]

Hi there,

First, we're using Carmine successfully in production for a while now, so thanks a lot!

Recently, Redis has become the main bottleneck in our application. We think Redis pipelining over multiple requests (GET/SET KEY commands mostly) will remove this bottleneck for now. To coordinate this batching we think core.async is a good fit. One option is to use core.async's thread macro to do this, but I have the feeling there is also some callback process going on in Carmine's code, which would make it suitable for use in normal go blocks. I haven't been able to find this though. Is my intuition wrong?

Thanks in advance, Jeroen

[ptaoussanis]

Hi Jeroen, you're very welcome.

We think Redis pipelining over multiple requests (GET/SET KEY commands mostly) will remove this bottleneck for now.

Sure, good pipelining can make a significant difference to your max Redis ops/sec.

To coordinate this batching we think core.async is a good fit.

Here I'm losing you. What do you mean by "coordinate" exactly? Redis (+ Carmine) pipelining shouldn't need any particular coordination.

The point of pipelining is to reduce the effect of TCP overhead. Compare:

1. 3x PING ops with 3 separate req+response roundtrips.
2. 3x PING ops with a single pipelined req+response roundtrip.

Simple Clojure destructuring makes reading pipeline responses very easy.

which would make it suitable for use in normal go blocks

Again, not quite sure what you have in mind here - but I'd definitely try avoid heavy IO in your coroutines.

What specific problem are you trying to solve that makes you think core.async would be a good fit? Can you give an example?

[jeroenvandijk]

OK, apologies, now I re-read my post I see that I should have added more context.

I should have noted that we are already using pipelining. We batch all Redis commands belonging to one incoming http request. The average number of Redis commands per request is 20, but there are also requests where this is 10 or 100. I don't have precise metrics on this at the moment. We run with several servers at somewhere between 5k QPS and 80k QPS, so the load on Redis is always quite high. The CPU of the Redis server is not too high, though. Unfortunately, we see latencies in the Redis requests that are higher than what our requirements allow (we're doing realtime bidding). So, just one of the optimizations, is to optimize for the ideal batch size. Since we have a high frequency of requests that all need Redis data, we can combine the Redis batches of several requests into one batch without introducing too much (wait) latency. The easiest way of coordinating this for multiple requests, I can think of, is the use of core.async.

core.async gives us two options to do async operations, either via the go or via thread macro. thread is necessary when you want to wrap an API that is synchronous/blocking. go is more lightweight, but requires an asynchronous API to avoid blocking the underlying go thread pool. So I'm wondering if Carmine has an asynchronous API available under the hood that I could use.

Does the above make sense now?

[ptaoussanis]

Unfortunately, we see latencies in the Redis requests that are higher than what our requirements allow (we're doing realtime bidding)

Will depend on the environment + what you're doing, but note that Redis can often happily hit 100k-400k reqs/sec on server hardware with pipelining. And as you noted, that's often with minimal CPU use.

If you're seeing high latency, first thing I'd look at is what ops you're running - e.g. O(1) vs O(n), etc. This can be especially important on Lua scripts.

The easiest way of coordinating this for multiple requests, I can think of, is the use of core.async.

Yeah, sorry- still not too sure what you have in mind with this exactly. But so long as you're clear on the plan :-)

So I'm wondering if Carmine has an asynchronous API available under the hood that I could use.

Sorry, asynchronous in what way? Maybe you can give me an example of what you'd see as an asynchronous Redis API call?

[jeroenvandijk]

I must be wanting something very specific, sorry it felt as a common thing. As an example for asynchronous vs synchronous, let's use httpkit vs clj-http. For clj-http to be compatible with core.async i would have do something like this:

   (require '[clojure.async :as a])

   (defn async-clj-http-request []
     (let [ch (a/chan 1)]
       (thread      
         (let [response (clj-http-client/get "http://site.com/resources/id")]
           (a/>!! ch response)))
       ch))

Whereas httpkit allows you to use a callback that can be used to fill a channel:

  (defn async-httpkit-request []
    (let [ch (a/chan 1)]
      (httpkit-client/get "http://site.com/resources/id" #(a/put! ch %))
       ch))

The idea here is that the httpkit allows for more concurrency and occupies less threads (also see this blogpost about it). I was hoping that Carmine would have some similar kind of callback structure underneath too, so I could apply the same trick as with httpkit. My expectation would be that it gives me the same benefit, that of a lower thread usage.

I don't know how to explain myself further, but if this all doesn't make sense to you I will conclude there is only a synchronous API. In that case I'll go ahead with the thread approach and see how that goes.

Either way, I'll point you to a gist of my code when I have something that batches/pipelines over multiple requests.

Thanks, Jeroen

[ptaoussanis]

I was hoping that Carmine would have some similar kind of callback structure underneath too, so I could apply the same trick as with httpkit

Yeah, no - sorry: Redis is essentially synchronous TCP req->response. Now technically you can batch (pipeline) requests without actually waiting on a response (Carmine lets you do this too)- but really don't think you'll see any benefit from fiddling with something like that in practice.

Redis (and Carmine) are orders of magnitude faster than, say, an HTTP request - so any coordination/core.async overhead is going to completely dominate your timings. Something like core.async would likely be much slower than just waiting on a response.

Again, exception here may be if you're running stuff like the KEYS command in production which is O(n) | n is the size of your entire keyspace. Keep in mind that Redis is single threaded as far as ops are concerned.

Have you checked your Redis SLOWLOG?

[jeroenvandijk]

No I haven't digged into SLOWLOG yet, because we are just doing GET's and SET's, but I think you are right and I should.

And you are probably also right about the TCP speed, I might just get away with using core.async's pipeline-blocking with a specific n that reduces the core.async overhead, but allows for enough concurrency at the same time.

Thanks a lot for your time! I'll let you know what I find.

Jeroen

[ptaoussanis]

because we are just doing GET's and SET's

Okay if that's the case, then the slowlog won't be of much help. If you're already maxing out your hardware's GET/SET capacity for one Redis server, then your only real options are:

1. More aggressive Redis command pipelining
2. Better hardware
3. Try run multiple Redis servers on the same hardware (may or may not be useful depending on the particular cause of your bottleneck)

Tbh really can't think of anything that core.async would be helpful with here, but it's possible I'm still misunderstanding something about your architecture or goals.

Thanks a lot for your time! I'll let you know what I find.

Sure, no problem- good luck! :-)