search

eddiewebb / circleci-queue

CircleCI orb to block/queue jobs to enforce max concurrency limits
MIT License
74 stars 75 forks source link

support for serializing workflows by name #76

Closed ashanbrown closed 8 months ago

ashanbrown commented 2 years ago

Is your feature request related to a problem? Please describe. I'd like to serialize a workflow of a specific name without other workflows preventing it from running. Currently we wait for all workflows to complete before starting a workflow.

Describe the solution you'd like Can we check only for existing workflows by workflow name (similar to what we do for serializing jobs)?

Describe alternatives you've considered None

Additional context

chabou commented 2 years ago

I need this same feature. @eddiewebb Would you accept a PR for this?

DReigada commented 2 years ago

I would also like to have this feature.

Would be up for doing the change

eddiewebb commented 2 years ago

@chabou @DReigada

I would happily review pr with this feature. Please include tests that validate.

chabou commented 2 years ago

Can we discuss about the right way to activate/configure this feature? I was thinking to add an optional parameter like consider-workflow-name (false by default to be backward compatible).

DReigada commented 2 years ago

@chabou I was thinking something similar to that. But I was wondering how the parameter block-workflow would behave if a new parameter like that is introduced.

Some thoughts (when consider-workflow-name is true):

It seems that ideally this would be a single parameter with 3 possible values: "block by job name", "block by workflow name", "block by anything".

@eddiewebb Any thoughts how this should be implemented?

chabou commented 2 years ago

I was thinking:

This is not ideal but has a better backward compatibility than introducing a multiple values parameter. I don't know what is the best

chabou commented 2 years ago

Maybe it could be useful to consider "only in a workflow with the same name" even for job names filtering (ie. block-workflow is false). It would be simple to explain.

eddiewebb commented 2 years ago

It may be a breaking change and major version rev, but rather than overlapping parameters we move to a single regex which can be empty, everything ('*'), or specific name.

I think that's how we ended up with job name filtering.

Use of old parameter could by-pass new logic for a ring and serve deprecation notice, or could just fail build and explain new model.

Appreciate the dialog on this.

On Fri, Aug 19, 2022, 11:01 AM CHaBou @.***> wrote:

Maybe it could be useful to consider "only in a workflow with the same name" even for job names filtering (ie. block-workflow is false). It would be simple to explain.

— Reply to this email directly, view it on GitHub https://github.com/eddiewebb/circleci-queue/issues/76#issuecomment-1220781551, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIEWSVN5PZSTLC2A66A5ITVZ6OS5ANCNFSM5PVRKZIQ . You are receiving this because you were mentioned.Message ID: @.***>

chabou commented 2 years ago

Yes you're right, regex is the most powerful solution.

Deprecatation warning and automatically transforming block-workflow: true into * value for the new parameter seems perfect.

workflow-regex is a perfect name I guess.

Do we agree that this new parameter is used in addition of other filtering parameter like only-on-branch or job-regex? block-workflow is a parameter but a job too. Should we deprecate this job too and keep only queue/until_front_of_line command with some documentation to explain how to have a first job with this command to block an entire workflow?

chabou commented 2 years ago

If it used in combination with job-regex, should it be considered like an AND or an OR condition? I think that is a AND: the job will be queued only if there is a job with a name matching job-regex AND in a workflow with a name matching workflow-regex. But in this case, without a job-regex parameter, should workflow-regex disable the default behavior of matching the job name (to behave like the deprecated block-workflow)?

eddiewebb commented 2 years ago

I think more and condition. I don't like the idea of dropping the job.

Empty workflow means don't block workflow level, and consider job logic. An asterisk means all workflows will block, but should consider job logic. Similar for named or pattern.

Easiest maybe to enumerate use cases and how to configure.

1) regardless of job make all workflows serialize 2) regardless of activity(job) serialize this specific workflow 3) regardless of workflow, serialize this job. 4) on a specific branch, serialize this job

... Branch logic should be completely independent and the primary decision.

On Sat, Aug 20, 2022, 4:35 AM CHaBou @.***> wrote:

If it used in combination with job-regex, should it be considered like an AND or an OR condition? I think that is a AND: the job will be queued only if there is a job with a name matching job-regex AND in a workflow with a name matching workflow-regex. But in this case, without a job-regex parameter, should workflow-regex disable the default behavior of matching the job name (to behave like the deprecated block-workflow)?

— Reply to this email directly, view it on GitHub https://github.com/eddiewebb/circleci-queue/issues/76#issuecomment-1221259116, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIEWSVV7XXEDJ7OPLMT5VLV2CKEHANCNFSM5PVRKZIQ . You are receiving this because you were mentioned.Message ID: @.***>

chabou commented 2 years ago

I would suggest these configurations:

  1. regardless of job make all workflows serialize job-regex: empty workflow-regex: .*
  2. regardless of activity(job) serialize this specific workflow job-regex: empty workflow-regex:
  3. regardless of workflow, serialize this job. job-regex: workflow-regex: empty
  4. on a specific branch, serialize this job job-regex: workflow-regex: empty only-on-branch: 5) serialize this job only if it belongs to this specific workflow job-regex: workflow-regex:

The job block-workflow would be an alias of case 1: job-regex: empty workflow-regex: .*

chabou commented 2 years ago
  1. serialize job with the same name (default): job-regex: empty workflow-regex: empty
valernyx commented 2 years ago

Just chiming in here to offer that we are looking for a feature in our organization to serialize a specific job when it's called in a specific workflow, but not serialize it across all workflows.

We currently have the queue/until_front_of_line enabled for our job that must be serialized but it queues it across all workflows and we only really want to queue it so it can only run one instance of the job within each workflow at a time.

Because we have a large number of workflows, we are getting some very long queues without the ability to do this. I love this orb, and would 100% support a feature to restrict the serialization to within the a workflow, vs. across all. What our ideal state would be is across (5) workflows, only (1) of the job-to-serialize could be running in each workflow at each time, so that it would allow (5) job-to-serialize to run at once, as long as each one was running in a separate workflow.

chabou commented 2 years ago

@valernyx I think your usage is the case 5. Will it feet your needs to specify job's name and workflow's name in each workflow (five time in your exemple)?

valernyx commented 2 years ago

@chabou Yes, I believe case 5 would support our needs. It sounds like I would be able to specify the job-to-serialize in each workflow and specify the workflow name to serialize it in as well, so as long as it would allow the job to run (1) time in each workflow (while supporting concurrency across workflows), then this sounds perfect 👍

eddiewebb commented 8 months ago

93 resolved this