search

saurabhnanda / odd-jobs

Haskell job queue with admin UI and loads of other features.
https://www.haskelltutorials.com/odd-jobs/
BSD 3-Clause "New" or "Revised" License
75 stars 29 forks source link

Add resource-based concurrency control #80

Closed ivb-supercede closed 1 year ago

ivb-supercede commented 3 years ago

This is based on the system in saurabhnanda/odd-jobs#55, and aims to address saurabhnanda/odd-jobs#38, where jobs need to have some limited access to resources that controls how many can run simultaneously.

Unlike that PR, this implements a system where jobs can require access to 0 or more resources, with different amounts of usage. This is because of a business need for jobs to be able to require multiple resources.

The implementation is intended to have no performance impact on existing code wherever the user has not opted in to resource-based concurrency. This is done by having parallel implementations wherever necessary that switch based on the concurrency control chosen.

ivb-supercede commented 3 years ago

I apologise for rewriting this resource system from scratch, instead of adapting the code from the other PR, since it creates additional reviewing load. But we really need a many-to-many job-to-resource relation, and the use of ResourceCfg also neatly solves the performance concerns in my eyes by making the extra tables & query logic completely opt-in.

jappeace commented 3 years ago

@saurabhnanda could you please look at this?

ivb-supercede commented 1 year ago

I'm going to merge this, since it would be good to cut a major release soon, and this is an often-requested feature that works well for us at Supercede.