Message queue sometimes stop processing messages

[apa512]

Often my worker stops processing jobs after a while. No error message or anything, just keeps on running but without processing jobs. I have quite a heavy workload with 1000000 jobs that take about 5 seconds each running with 160 nthreads. I tried with as few as 20 threads but the problem remained. Any tips on how to debug this? Nothing happens for several hours so I doubt it's somehow related to backoff.

Here's my code, which I'm running with lein run:

(defn index-bucket [{:keys [mid message attempt]}]
  (println "Working on" message)
  (time
    (do-things)
    {:status :success})))

(def index-bucket-worker
  (car-mq/worker
    redis-conn
    "index-bucket"
    {:handler index-bucket
     :nthreads 160
     :auto-start false}))

(defn -main []
  (.addShutdownHook (Runtime/getRuntime) (Thread. #(stop-worker)))
  (start-worker))

[ptaoussanis]

Hi Erik, please see the worker fn's :monitor option as one possible monitoring/logging/debug hook. The use is documented in the worker docstring.

Let me know if that helps?

Cheers! :-)

[apa512]

Thanks. I will try that next time. I also remembered something else that may be of importance. The proposed way of adding messages was too slow for adding one million jobs so I did this instead. Bad idea perhaps?

(wcar redis-conn
  (mapv
    #(car/hset "carmine:mq:index-bucket:messages" (str %) {:bucket %})
    (take 1000001 (range)))
    (mapv
      #(car/lpush "carmine:mq:index-bucket:mid-circle" (str %))
      (take 1000001 (range))))

[ptaoussanis]

Closing this assuming you've found the cause of your trouble. Feel free to reopen if you're still having issues.