Enable configurable queue concurrency at project level

[christad92]

Configurable concurrency is a mechanism that allows us to restrict how many workflows/jobs the worker can run in parallel within a project. When concurrency is set to 1, only a work order/run/job can be processed at a time and other jobs will be enqueued until that is done.

How this worked in v1

openfn often will…
1. get bunch of csv files from sftp --> convert to JSON // Input data 
2. post json payloads to project Inbox (but post the payloads in the order we want to process -e.g., first post Contacts 
data and then related Donations data bc Donations can’t exist without Contacts)
3. OpenFn would then process all Contacts messages 1-at-a-time first… before then moving onto the Donations message (moving chronologically through the Inbox/Runs queue)

TODOs:

• [ ] Artificially limit (or increase) the number of runs being executed at once per project
• [ ] Enable users to add a cool-off period between runs in a project

Acceptance Criteria

• [ ] A user can configure concurrency and cool-off period in the project settings as integer inputs.
• [ ] When the concurrency is set to X, only X run(s) can be executed at a time, every extra run requested is enqueued
• [ ] Runs should not be executed between cool-off periods but run requests will be enqueued

Questions:

• What is the current behavior? Do we allow multiple jobs to run concurrently?
• Should we limit the number of concurrency a user can set?
• Should we notify the project owner or admins when a threshold is reached as in v1?
• Should this threshold be based on the queue or number of concurrent runs relative to the project maximum? There is an opportunity to upsell an upgrade if the user has a long queue and needs to clear them faster.

[taylordowns2000]

I'd say we need two things here, almost certainly separate issues. Something around: https://github.com/OpenFn/lightning/blob/main/lib/lightning/runs/queue.ex#L12

1. Artificially limit (or increase?) the number of runs being excecuted at once per project (concurrency dial, just like v1). Something like...?

   i'm the worker, give me runs to execute...
   WHERE NOT IN runs.state == executing AND runs.project_id IN (
   some sort of list of projects?
   )

2. Artificially add a "cool-down" to stall, per project, after each run. Something like...?

   i'm the worker, give me runs to execute...
   WHERE NOT IN min(age(now(), runs.finished_at)) > '10 seconds' AND runs.project_id IN (
   some sort of list of projects?
   )

Nice blog that @stuartc found - https://docs.hatchet.run/blog/multi-tenant-queues#introducing-concurrency-limits

And here's the UI from v1:

[christad92]

Stu is yet to spike the approach to implementing this. cc: @stuartc