mosquito-cr / mosquito

A background task runner for crystal applications supporting periodic (CRON) and manually queued jobs
MIT License
227 stars 24 forks source link

provides for multiple executors operating under a single runner #123

Closed robacarp closed 11 months ago

robacarp commented 1 year ago

This will fix #122 #18 and #80

My goal here is to minimally make mosquito network bound workloads significantly faster. Testing with @mamantoha shards.info codebase it’s trivial to cut the scrape time for the whole run by 60-70% because the whole workload is network bound. CPU abound workloads will have almost no benefit here, but the framework is laid for multi process work in the future.

As always, comments welcome and appreciated.