eddiewebb / circleci-queue

CircleCI orb to block/queue jobs to enforce max concurrency limits
MIT License
74 stars 75 forks source link

[3.1.4] How to block workflows that are triggered with commits and tags #130

Open ArthurKnoep opened 1 month ago

ArthurKnoep commented 1 month ago

Orb version

3.1.4

What happened

I'm working on a project that defines two main CircleCI workflows:

The issue I have is that the building, deploying and testing of the end to end tests must not ran concurrently, whatever the workflow.

For the block_e2e job of the "commit" workflow I'm using the following definition

      - queue/block_workflow:
          name: block_e2e
          max-wait-time: '180'
          this-branch-only: false
          job-regex: '^build_deploy_e2e|run_e2e$'
          my-pipeline: << pipeline.number >>
          confidence: '2'

For the block_e2e job of the "tag" workflow, I'm using the same definition but I'm adding a filters defintion to trigger the job in CircleCI and a tag-pattern:

      - queue/block_workflow:
          filters:
            tags:
              only: /^v.*$/
            branches:
              ignore: /.*/
          name: block_e2e
          tag-pattern: '^v.*$'
          max-wait-time: '180'
          this-branch-only: false
          job-regex: '^build_deploy_e2e|run_e2e$'
          my-pipeline: << pipeline.number >>
          confidence: '2'

The issue I'm having is that by adding the tag-pattern variable it ignores the workflow running for the "commit" workflow (at least from what I understand of this line

But if I don't specify the tag-pattern parameter the queue blocker won't run, because of this condition

Expected behavior

I would like to have my workflow blocked according to every workflow (both triggered by tags or commits)

eddiewebb commented 1 month ago

I might not be fully aligned, but if you want the workflow to consider any matching workflow, tagged or commit, then don't specify a tag pattern. You are telling it to ignore commit (and other sources of change) by specifying it.

The job regex you have should cover the screenshot you described(I think)

On Wed, May 29, 2024, 6:17 AM Arthur Knoepflin @.***> wrote:

Orb version

3.1.4 What happened

I'm working on a project that defines two main CircleCI workflows:

  • One is triggered on every commit on the main branch for staging and run something similar to this:
    • [block_e2e] -> [build_deploy_e2e] -> [run_e2e] -> [block_deploy_stg] -> [build_stg] -> [deploy_stg]
  • One is triggered on tags on the main branch and run something very much similar but deploys on production instead:
    • [block_e2e] -> [build_deploy_e2e] -> [run_e2e] -> [block_deploy_prod] -> [build_prod] -> [deploy_prod]

The issue I have is that the building, deploying and testing of the end to end tests must not ran concurrently, whatever the workflow.

For the block_e2e job of the "commit" workflow I'm using the following definition

  - queue/block_workflow:
      name: block_e2e
      max-wait-time: '180'
      this-branch-only: false
      job-regex: '^build_deploy_e2e|run_e2e$'
      my-pipeline: << pipeline.number >>
      confidence: '2'

For the block_e2e job of the "tag" workflow, I'm using the same definition but I'm adding a filters defintion to trigger the job in CircleCI and a tag-pattern:

  - queue/block_workflow:
      filters:
        tags:
          only: /^v.*$/
        branches:
          ignore: /.*/
      name: block_e2e
      tag-pattern: '^v.*$'
      max-wait-time: '180'
      this-branch-only: false
      job-regex: '^build_deploy_e2e|run_e2e$'
      my-pipeline: << pipeline.number >>
      confidence: '2'

The issue I'm having is that by adding the tag-pattern variable it ignores the workflow running for the "commit" workflow (at least from what I understand of this line https://github.com/eddiewebb/circleci-queue/blob/edfb701b59d73a64dcc4c94af6d42ed4a3fe1f9b/scripts/loop.bash#L139

But if I don't specify the tag-pattern parameter the queue blocker won't run, because of this condition https://github.com/eddiewebb/circleci-queue/blob/edfb701b59d73a64dcc4c94af6d42ed4a3fe1f9b/scripts/loop.bash#L54 Expected behavior

I would like to have my workflow blocked according to every workflow (both triggered by tags or commits)

— Reply to this email directly, view it on GitHub https://github.com/eddiewebb/circleci-queue/issues/130, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIEWSXYXOY4AOYCO4D765DZEXBOFAVCNFSM6AAAAABIO2DWA6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGMZDGMJXHA2TANQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

eddiewebb commented 1 month ago

Should satisfy the scenario you described.

On Thu, May 30, 2024, 7:12 PM Edward A. Webb @.***> wrote:

I might not be fully aligned, but if you want the workflow to consider any matching workflow, tagged or commit, then don't specify a tag pattern. You are telling it to ignore commit (and other sources of change) by specifying it.

The job regex you have should cover the screenshot you described(I think)

On Wed, May 29, 2024, 6:17 AM Arthur Knoepflin @.***> wrote:

Orb version

3.1.4 What happened

I'm working on a project that defines two main CircleCI workflows:

  • One is triggered on every commit on the main branch for staging and run something similar to this:
    • [block_e2e] -> [build_deploy_e2e] -> [run_e2e] -> [block_deploy_stg] -> [build_stg] -> [deploy_stg]
  • One is triggered on tags on the main branch and run something very much similar but deploys on production instead:
    • [block_e2e] -> [build_deploy_e2e] -> [run_e2e] -> [block_deploy_prod] -> [build_prod] -> [deploy_prod]

The issue I have is that the building, deploying and testing of the end to end tests must not ran concurrently, whatever the workflow.

For the block_e2e job of the "commit" workflow I'm using the following definition

  - queue/block_workflow:
      name: block_e2e
      max-wait-time: '180'
      this-branch-only: false
      job-regex: '^build_deploy_e2e|run_e2e$'
      my-pipeline: << pipeline.number >>
      confidence: '2'

For the block_e2e job of the "tag" workflow, I'm using the same definition but I'm adding a filters defintion to trigger the job in CircleCI and a tag-pattern:

  - queue/block_workflow:
      filters:
        tags:
          only: /^v.*$/
        branches:
          ignore: /.*/
      name: block_e2e
      tag-pattern: '^v.*$'
      max-wait-time: '180'
      this-branch-only: false
      job-regex: '^build_deploy_e2e|run_e2e$'
      my-pipeline: << pipeline.number >>
      confidence: '2'

The issue I'm having is that by adding the tag-pattern variable it ignores the workflow running for the "commit" workflow (at least from what I understand of this line https://github.com/eddiewebb/circleci-queue/blob/edfb701b59d73a64dcc4c94af6d42ed4a3fe1f9b/scripts/loop.bash#L139

But if I don't specify the tag-pattern parameter the queue blocker won't run, because of this condition https://github.com/eddiewebb/circleci-queue/blob/edfb701b59d73a64dcc4c94af6d42ed4a3fe1f9b/scripts/loop.bash#L54 Expected behavior

I would like to have my workflow blocked according to every workflow (both triggered by tags or commits)

— Reply to this email directly, view it on GitHub https://github.com/eddiewebb/circleci-queue/issues/130, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIEWSXYXOY4AOYCO4D765DZEXBOFAVCNFSM6AAAAABIO2DWA6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGMZDGMJXHA2TANQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

ArthurKnoep commented 3 weeks ago

Hi, thanks for your answer.

But if I do not specify the tag-pattern the queue just doesn't run on tag trigger (production workflow). I get this message in the logs:

TAG_PATTERN defined, but not on tagged run, skip queueing!

Which seems to come from this condition: https://github.com/eddiewebb/circleci-queue/blob/edfb701b59d73a64dcc4c94af6d42ed4a3fe1f9b/scripts/loop.bash#L54

eddiewebb commented 3 weeks ago

I see ..

We can drop that check and see if all use cases in bats test still pass.

Desired state is that supplying a tag will only concern itself with tagged builds, but omission will run on all, including tagged.

That sound right?

On Tue, Jun 4, 2024, 3:44 AM Arthur Knoepflin @.***> wrote:

Hi, thanks for your answer.

But if I do not specify the tag-pattern the queue just doesn't run on tag trigger (production workflow). I get this message in the logs:

TAG_PATTERN defined, but not on tagged run, skip queueing!

Which seems to come from this condition: https://github.com/eddiewebb/circleci-queue/blob/edfb701b59d73a64dcc4c94af6d42ed4a3fe1f9b/scripts/loop.bash#L54

— Reply to this email directly, view it on GitHub https://github.com/eddiewebb/circleci-queue/issues/130#issuecomment-2146834309, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIEWSWI7XN5ODYAIJTWDRLZFVV6FAVCNFSM6AAAAABIO2DWA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBWHAZTIMZQHE . You are receiving this because you commented.Message ID: @.***>