Closed ivb-supercede closed 3 years ago
Thank your for your PR. I'll take a little longer to review this PR, while #70 seems straightforward and should get merged as soon as the CI build passes.
In the meantime, is the use-case for your PR the same as the discussion at #38 and the subsequent PR at #55 ?
In the meantime, is the use-case for your PR the same as the discussion at #38 and the subsequent PR at #55 ?
The use-case is very similar - but we don't have any shared resource that jobs need to control access to (apart from system CPU & memory). Being able to rate-limit jobs more generally would also satisfy our needs.
@ivb-supercede would it fair to say that if we have concurrency control based on a resources
table (as implemented in #55) it can satisfy your use-case as well, by treating resource = job-type. But it will require more work on your behalf to set up the resources
table appropriately.
I'm wondering if it makes sense to make the jobPollingSql
configurable and give a bunch of common options to choose from (with a sensible default in ConfigBuilder.mkConfig
)
Closing in favour of #80.
This is an important use case for us, since we have certain expensive jobs that shouldn't hog all the resources on the machine. This adds a new kind of ConcurrencyControl, that limits the number of jobs that can be simultaneously executed per job-type.
I tried to modify the code in such a way that polling is not made significantly more expensive by the existence of the new concurrency control - but there might be a slight decrease in performance, because the job type filter is filled from the list of all job types, which might result in a query. This could be fixed by making two polling queries: one which takes the filter, and one which doesn't.