search

edgurgel / verk

A job processing system that just verks! 🧛‍
https://hex.pm/packages/verk
MIT License
721 stars 65 forks source link

feedback; Configuration examples/feedback for systems with limited resources (memory)? #164

Closed florinpatrascu closed 6 years ago

florinpatrascu commented 6 years ago

Hi there, and thank you for this project!

I'd like to learn if any of the Verk users here are running systems where they encountered situations where the number of queued jobs are accumulating at a rate such that they might exceed the total memory available to Redis?! What happens in these situations? Any advices you could share? I'd avoid clustering Redis.

Thank you!

edgurgel commented 6 years ago

Hey @florinpatrascu ,

I can give you my view on this topic. Verk is really useful if you want to have a set of queues processing jobs and having specific limits per instance for each queue. But it's not meant to be used like Kafka or Rabbitmq consumers. It won't use disk to store excess of jobs for example.

Verk is also capable of scheduling jobs to run in the future and also a sensible retry strategy that also respects the time when it should retry. All of this is not "trivial" to achieve using "dumb" Kafka/Rabbitmq/ActiveMQ consumers.

What we do is: we load test our instances so that we know what's the capacity of memory we should have and how long it would hold with a slow processing. And we set up alarms for the memory usage so that we know what we need to do. We also have alarms for queue size which you can get from Verk.Queue.count/1) for example.

We are also investigating ways of getting better metrics like "queue latency" which is: how long it's taking since we enqueued a message to the moment we finished it so that we can set up alarms.

florinpatrascu commented 6 years ago

Thank you so much, Eduardo! It's inline with what we were hoping to hear, and the extra details about the criteria for setting alarms, was a boon! Thank you so much, and thank you for Verk, once again :)

edgurgel commented 6 years ago

@florinpatrascu , no problem!

BTW I'm also working on a project called verk-stats which basically extracts this code that publishes metrics about the queues/jobs and it will include the queue/job latency so everybody can use it. It can be used with statsd or anything really. You will just need to define a "sink" pretty much like vmstats does.