Setting activityGap per "kind" of job?

[hems]

On my application some jobs ( or type of jobs ) can have a different activityGaps, so i believe it would be important to have activeGap set per document and not as a "global" option if that makes sense?

[msavin]

This is interesting - most are leaning towards running jobs simultaneously but this goes the opposite direction. Can you please tell me more about the use case?

[hems]

Context :

• many workers ( say 5-N )
• different type of jobs ( 3-8 )

Rules:

• All workers are entitled to work any type of job
• All workers can work simultaneously in different jobs
• Each type of job can have a different ActivityGap
• If a job fails to be finished ( for instance a worker crash ) it should not reset it activity gap so another worker can try to work the same job

So the way i see it working is:

1. Worker looks for the job ordered by jobs waiting longer
2. Worker timestamp jobs as "being worked" ( so no other worker can work this job )
3. When job is finished, job is marked as "worked" and is available again on the queue

The only difference here in relation to what you have done is the idea that each type of job can have a different ActivityGap, for instance if a job involves calling a rate limited API we don't want to do it more often than the API would allow us to while jobs that have no rate limit can be performed more often.

Does that make sense ?

[msavin]

@hems I'm not sure the package works in this way.

As for situations like rate limiting, the package now supports this.reschedule() to delay the job, and you can also set state for each job with this.set and this.get. Right now, the best I can suggest is carefully scheduling jobs to make sure you do not have too many running one after another.

[hems]

thanks @msavin !