Open pss4281 opened 7 years ago
Hi, thanks for the question. job-collection is not a "real-time" system, in that it only makes a "best effort" to run/repeat/retry jobs when you ask them to. In your case you would like to maximize throughput among multiple workers without exceeding some hard rate-limit threshold. job-collection doesn't have a "built-in" way to do this.
What you need is some type of external sync mechanism that your workers can all access to get "rate permission" to access the external service.
It could be something as simple as a Meteor method call to the server that accesses an in-memory state variable that is set to (say) 10, once per second, by a Meteor.setInterval()
callback. Each call to the method would decrement the state variable and return true
if it is > 0, or false
otherwise. Because node.js is single threaded, no fancy synchronization is required when accessing that variable on the server.
@vsivsi thanks for a prompt reply! the setup you described (countdown quotas variable that gets refreshed every second) might work! I'm curious about how would the jobWorker skip it's turn if variable is === 0? We can fail the job in that case and hope that it will re-run in next second. But hope is kinda a bad thing to develop around :)
appreciate your answers
I wouldn't have the worker fail. I'd simply sleep it (with setTimeout()
) for some amount of time (500ms?) and have it try again when it wakes up.
thanks again @vsivsi this seems to be workable. Just need to fine-tune this sleep timeout to ensure they don't wait unnecessarily long.
Hello,
got a question here. Let's suppose there is a use case like this:
so the question is how to insure proper rate (so that we don't make more than allowed number of requests/second i.e. we don't run more than allowed number of jobs/second) and keep the efficiency at a good level (e.g. if 10 req/second is allowed we do 10 and not 1 or 2)
I assume one of the options is to set each job's
after
field depending on prev. job'safter
? Any better way to tackle this?thanks!