search

mdawar / rq-exporter

Prometheus metrics exporter for Python RQ (Redis Queue).
MIT License
65 stars 28 forks source link

Kubernetes - Horizontal POD auto scaling #10

Closed vijtad closed 3 years ago

vijtad commented 3 years ago

Can any one give me an example of how this can be used to export length of queue and autoscale it.

I want to scale workers when the queue size is more than 25.

mdawar commented 3 years ago

Hi,

Sorry I can't help you with Kubernetes, but you should check the rq_jobs metric to get the number of jobs by status per queue:

rq_jobs{queue="default", status="queued"} 2.0
rq_jobs{queue="default", status="started"} 1.0
rq_jobs{queue="default", status="finished"} 5.0
rq_jobs{queue="default", status="failed"} 1.0
rq_jobs{queue="default", status="deferred"} 1.0
rq_jobs{queue="default", status="scheduled"} 2.0

A simple search on Google leads us to a couple of blog posts about metrics based autoscaling, here are some these posts: https://www.metricfire.com/blog/prometheus-metrics-based-autoscaling-in-kubernetes/ https://appfleet.com/blog/prometheus-metrics-based-autoscaling-in-kubernetes/

I hope that helps.