QOS style with priorities job processing - maybe Aging?

[gVolop]

@vsivsi, hi!

i have many types of out http requests, and i put them all in one queue. but some of them i'll give an high priority of the others.

what will be if i have a case with many jobs from first priority? this will completely blocks all the others jobs until they all completed?

or anyway the others priorities will be handled in some time periods (or another way..) maybe aging options? wikipedia-aging Aging is used to gradually increase the priority of a task, based on its waiting time in the ready queue Many thanks for the awesome package!

[vsivsi]

Hi, job-collection has a relatively simple priority scheme, and aging as you describe is not implemented (because there are many such possible schemes, and covering all possible use-cases would be complex and probably ultimately infeasible).

However, the beauty of job-collection is that many such custom tasks can be added to the system by simply creating a job to implement them.

It should be straightforward for you to implement your own custom aging scheme by creating a repeating job on the server that queries for other jobs that meet some criteria you define, and then promotes them to higher priorities by pausing each job, changing its priority, and then saving the changes.

[gVolop]

yes. i thought about this solution. but if we can add a QOS functionality into Job-collection it can be awesome. just to continue to give service also for low level's jobs even we have a many high priority jobs. so after X jobs from high priority - processing one job from the lower (just let the programer to configure the frequency)

i thought about it for case i have some jobs that do same work (http requests, with different parameters (url, options...)), but with other parameters. so i want to give them another priority... but Okay, i export each type of request to another queue, and i use one function as worker. and rather than use priority i just configure the concurrency..

[vsivsi]

You can effectively do this by separating tasks by type and assigning a different mix of workers with different capabilities.

For example, if I want job type A to have 10:1 priority over job type B, I can simply create 10 workers that only process A jobs for every 1 worker that processes only B jobs. If you are concerned about maximizing utilization, you can enhance that by making the A workers capable of processing both A and B jobs, but schedule A jobs at a higher priority. However, the minority "B only" worker(s) will ensure that your QoS objectives are met with respect to B jobs.

Anyway, there are many possible ways to do this with the existing system. I would be hesitant to add QoS functionality to the core of this package without fairly broad demand for the functionality; and a clear best (general, simple and efficient) way to do it. I am a big believer in the 80/20 rule. I only want to add features that many/most users will eventually find a use for, and when added, any such feature should cover at least 80% of all use cases.

Given that you are the first developer to request this feature over the past two years, I'm skeptical that most applications would benefit from it. And for those that would, I would need to be convinced that there is a single simple feature that could be added to cover the vast majority of those use cases, without significantly degrading the performance of applications that don't require the functionality.

Anyway, I'm pretty confident you can implement precisely the behavior your application requires using the current features in at most a few dozen lines of code.

[gVolop]

ok. you are right. and that was the solution i started yesterday. trying to play with existing rules and options to get my purpose

thanks a lot!