search

MeVitae / redis-work-queue

A cross-platform work queue, on top of a redis database, with implementations in Python, Rust, Go, Node.js (TypeScript) and Dotnet (C#).
MIT License
15 stars 2 forks source link

Don't add duplicate jobs #3

Open JOT85 opened 1 year ago

JOT85 commented 1 year ago

When spawning jobs, is possible that the same job may be requested multiple times. An API could check if it already has the results and, if not, spawn a job. This approach could result in the job being spawned multiple times if it's already in the queue but not completed when it's requested again. We currently deal with this by having each worker check that the job it receives hasn't already been completed, and returning immediately if it has been.

Perhaps a neater approach is to add the job to the queue only if it isn't already in the queue or being processed.

Users can use a hash of the job to generate an ID for it, therefore, to achieve this, we need a method to add an item only if there isn't an item with the same ID already in the queue.

JOT85 commented 1 year ago

Also, a really important note is that duplicated jobs mess around with autoscaling, since they'll complete instantly so shouldn't really be counted! Not having duplicated jobs will increase the effectiveness of the autoscaler.