search

vsivsi / meteor-job-collection

A persistent and reactive job queue for Meteor, supporting distributed workers that can run anywhere.
https://atmospherejs.com/vsivsi/job-collection
Other
388 stars 68 forks source link

What is the best way to ensure job execution rate? #209

Open pss4281 opened 7 years ago

pss4281 commented 7 years ago

Hello,

got a question here. Let's suppose there is a use case like this:

so the question is how to insure proper rate (so that we don't make more than allowed number of requests/second i.e. we don't run more than allowed number of jobs/second) and keep the efficiency at a good level (e.g. if 10 req/second is allowed we do 10 and not 1 or 2)

I assume one of the options is to set each job's after field depending on prev. job's after? Any better way to tackle this?

thanks!

vsivsi commented 7 years ago

Hi, thanks for the question. job-collection is not a "real-time" system, in that it only makes a "best effort" to run/repeat/retry jobs when you ask them to. In your case you would like to maximize throughput among multiple workers without exceeding some hard rate-limit threshold. job-collection doesn't have a "built-in" way to do this.

What you need is some type of external sync mechanism that your workers can all access to get "rate permission" to access the external service.

It could be something as simple as a Meteor method call to the server that accesses an in-memory state variable that is set to (say) 10, once per second, by a Meteor.setInterval() callback. Each call to the method would decrement the state variable and return true if it is > 0, or false otherwise. Because node.js is single threaded, no fancy synchronization is required when accessing that variable on the server.

pss4281 commented 7 years ago

@vsivsi thanks for a prompt reply! the setup you described (countdown quotas variable that gets refreshed every second) might work! I'm curious about how would the jobWorker skip it's turn if variable is === 0? We can fail the job in that case and hope that it will re-run in next second. But hope is kinda a bad thing to develop around :)

appreciate your answers

vsivsi commented 7 years ago

I wouldn't have the worker fail. I'd simply sleep it (with setTimeout()) for some amount of time (500ms?) and have it try again when it wakes up.

pss4281 commented 7 years ago

thanks again @vsivsi this seems to be workable. Just need to fine-tune this sleep timeout to ensure they don't wait unnecessarily long.