measuring speed of car-mq/worker

[jdkealy]

Hi,

I'm looking for clojure job enqueueing and this library looks very promising. I'd like to distribute amongst different servers -- like I've used Sidekiq in the past with Ruby.

I noticed you said the workers are very fast, was wondering what metrics you have on picking up the various jobs. The way I currently have my code set up, it's taking between 3 and 11 seconds. I assume it should be way faster, right ?

Code attached:

`(ns bfadam.utils.redis
  (:require
   [taoensso.carmine :as car :refer (wcar)]
   [me.raynes.conch :refer [programs with-programs let-programs] :as sh]
   [taoensso.carmine.message-queue :as car-mq]))

(def server-connection {:spec {:host "localhost"
                               :port 6379
                               :timeout 4000}})

(defmacro wcar* [& body] `(car/wcar server-connection ~@body))

(def timer (atom 0))

(defn get-time []
  (quot (System/currentTimeMillis) 1000))

(def my-worker
  (car-mq/worker server-connection "MQ1"
                 {:handler (fn [{:keys [message attempt]}]
                             (let [t (get-time )]
                               (println "Received" message (- t @timer )))
                             {:status :success})}))

(def my-worker2
  (car-mq/worker server-connection "MQ1"
                 {:handler (fn [{:keys [message attempt]}]
                             (let [t (get-time )]
                               (println "Received ON 2" message (- t @timer)))
                             {:status :success})}))

(comment
  (do
    (reset! timer (get-time))
    (wcar* (car-mq/enqueue "MQ1" "NUMBER OF SECONDS: "))))

bfadam.repl> 
Received NUMBER OF SECONDS:  7
Received NUMBER OF SECONDS:  6
Received ON 2 NUMBER OF SECONDS:  5
Received ON 2 NUMBER OF SECONDS:  3
Received NUMBER OF SECONDS:  11
bfadam.repl> 

`

[ptaoussanis]

Hi John,

I noticed you said the workers are very fast

Sorry, I've actually just removed that line from the README since it was misleading. Carmine's MQ throughput is reasonable but latency reasonable-to-slow (these comparared to other MQ's).

There'd be some obvious ways of improving the implementation's latency, but that's not something I need myself so hasn't been a priority. A future release may add Disque support which might deprecate Carmine's own queue (haven't looked closely at possible differences yet).

The way I currently have my code set up, it's taking between 3 and 11 seconds.

You'll want to look at the worker docstring for info on config options that'll affect your latency: :eoq-backoff-ms, :nthreads, and :throttle-ms.

The default :eoq-backoff-ms is optimized to minimize polling under high work variability.

Something like:

`{:eoq-backoff-ms (constantly 10) ; Back off 10ms when no work to do
  :throttle-ms 10 ; Wait 10ms between polling ops
  :nthreads _ ; Number of worker threads, up to you (tweak based on handling cost)
 }`

May be closer to what you'd like. Best of luck!

[jdkealy]

NOW we're cooking, THANKS!

`Received NUMBER OF SECONDS:  0
Received NUMBER OF SECONDS:  0
Received NUMBER OF SECONDS:  0
Received ON 2 NUMBER OF SECONDS:  0`

[ptaoussanis]

No problem, good luck! :-)