mosquito-cr / mosquito

A background task runner for crystal applications supporting periodic (CRON) and manually queued jobs
MIT License
227 stars 24 forks source link

Ability to throttle jobs #20

Closed Blacksmoke16 closed 5 years ago

Blacksmoke16 commented 6 years ago

Resolves #1

~However the more i think about it the one thing this wouldn't solve well is if the tasks get added slowly. For example you have a job that is set to 10 executions per 60 seconds. But you add one every 10min. This implementation would still would wait 60 seconds before executing the 11th task.~

This has been solved, it now only counts executions within the given period seconds.

How this works:

  1. I moved the @@mapping class var to a class_getter var, as i needed a way to get a list of all defined jobs
  2. When starting mosquito it will store a config hash into redis
  3. When a task is executed it will increment the executed field in the config object
  4. If Time.utc_now.epoch > (Time.epoch(config["last_executed"].to_i64) + config["period"].to_i.seconds).epoch resets executed back to 0
  5. When executed == limit, it will set the executed field back to 0 and set the next_batch field to (Time.utc_now + period.seconds).epoch
  6. When going to dequeue a task to run, it checks q_config["next_batch"].to_i64 > Time.utc_now.epoch and returns if true

WDYT @robacarp

Blacksmoke16 commented 6 years ago

~Noticing an issue where its doing 1 more execution than it should sometimes, i'll look into it.~

EDIT: Fixed in b4930ce

greenbigfrog commented 5 years ago

Any reason why this hasn't been merged yet?

Blacksmoke16 commented 5 years ago

IIRC I need to write tests for it, but was dependent on another shard being updated, that has since been forked and update.

I been using it for past few months and it's been fine as far as i can tell. However, not sure on the best way to test this, and hasn't been a high priority of mine lately :/

Blacksmoke16 commented 5 years ago

@robacarp This should be good for review again when you get some time.