search

film42 / sidekiq-rs

A port of sidekiq to rust using tokio
MIT License
95 stars 10 forks source link

Allow configuring the number of tasks spawned for workers #41

Closed spencewenski closed 2 months ago

spencewenski commented 2 months ago

Problem

If an app's workload is largely IO bound (querying a DB, making web requests and waiting for responses, etc), its workers will spend a large percentage of time idle awaiting for futures to complete. This in turn means the CPU will sit idle a large percentage of the time (if nothing else is running on the host), resulting in under-utilizing available CPU resources.

Solution

Allow apps to configure the number of tokio tasks that are spawned for workers. This will allow the app to configure the processor to use CPU resources more efficiently. For backwards compatiblity and in order to provide a sane/safe default, the default number of workers if not provided is the host's CPU count.

film42 commented 2 months ago

Looks good to me. I didn't impl this prior because I figured someone could copy the run run() method and change the number of concurrent workers. That said, this is really clean and I like it.

This is the one area I feel comfortable massaging the API until it's right so long as no functionality is removed once added.

film42 commented 2 months ago

Released as v0.10.3 Thanks again, @spencewenski ❤️