search

jenkinsci / amazon-ecs-plugin

Amazon EC2 Container Service Plugin for Jenkins
https://plugins.jenkins.io/amazon-ecs
MIT License
192 stars 226 forks source link

Maximum Agents is taken into account only if jobs are already running/agents are created, not if jobs are in the queue #269

Closed ibalosh closed 1 year ago

ibalosh commented 2 years ago

Jenkins and plugins versions report

Jenkins: 2.346.1 OS: Linux - 4.14.281-212.502.amzn2.x86_64

ace-editor:1.1 allure-jenkins-plugin:2.30.2 amazon-ecr:1.73.v741d474abe74 amazon-ecs:1.41 antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 artifact-manager-s3:633.v4813787e78a_9 authentication-tokens:1.4 authorize-project:1.4.0 aws-credentials:191.vcb_f183ce58b_9 aws-global-configuration:100.v4619b_45d2dfc aws-java-sdk:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-cloudformation:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-codebuild:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ec2:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ecr:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ecs:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-elasticbeanstalk:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-iam:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-logs:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-minimal:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-sns:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-sqs:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ssm:1.12.246-349.v96b_b_f7eb_a_c3c aws-secrets-manager-credentials-provider:1.2.0 blueocean:1.25.5 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.5 blueocean-commons:1.25.5 blueocean-config:1.25.5 blueocean-core-js:1.25.5 blueocean-dashboard:1.25.5 blueocean-display-url:2.4.1 blueocean-events:1.25.5 blueocean-git-pipeline:1.25.5 blueocean-github-pipeline:1.25.5 blueocean-i18n:1.25.5 blueocean-jwt:1.25.5 blueocean-personalization:1.25.5 blueocean-pipeline-api-impl:1.25.5 blueocean-pipeline-editor:1.25.5 blueocean-pipeline-scm-api:1.25.5 blueocean-rest:1.25.5 blueocean-rest-impl:1.25.5 blueocean-web:1.25.5 bootstrap4-api:4.6.0-5 bootstrap5-api:5.1.3-7 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 build-timeout:1.21 built-on-column:1.1 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.7.4 cloudbees-bitbucket-branch-source:773.v4b_9b_005b562b cloudbees-disk-usage-simple:0.10 cloudbees-folder:6.729.v2b_9d1a_74d673 command-launcher:84.v4a_97f2027398 commons-lang3-api:3.12.0.0 commons-text-api:1.9-9.v39a_53e2e0343 conditional-buildstep:1.4.2 configuration-as-code:1511.vb_f985b_894e40 copyartifact:1.46.4 credentials:1139.veb_9579fca33b credentials-binding:523.vd859a_4b_122e6 display-url-api:2.3.6 docker-commons:1.19 docker-java-api:3.2.13-37.vf3411c9828b9 docker-plugin:1.2.9 docker-workflow:1.29 durable-task:496.va67c6f9eefa7 echarts-api:5.3.3-1 email-ext:2.90 envinject:2.866.v5c0403e3d4df envinject-api:1.199.v3ce31253ed13 extended-read-permission:3.2 favorite:2.4.1 font-awesome-api:6.1.1-1 git:4.11.3 git-client:3.11.0 git-server:1.11 github:1.34.4 github-api:1.303-400.v35c2d8258028 github-branch-source:1656.v77eddb_b_e95df handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.30 jackson2-api:2.13.3-285.vc03c0256d517 jakarta-activation-api:2.0.0-3 jakarta-mail-api:2.0.0-6 javadoc:217.v905b_86277a2a javax-activation-api:1.2.0-4 javax-mail-api:1.6.2-7 jaxb:2.3.6-1 jdk-tool:1.5 jenkins-design-language:1.25.5 jenkins-multijob-plugin:611.v9d3180d752e6 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.15-1 jquery3-api:3.6.0-4 jsch:0.1.55.2 junit:1119.1121.vc43d0fc45561 ldap:2.10 lockable-resources:2.15 mailer:435.v79ef3972b_5c7 matrix-auth:3.1.5 matrix-project:772.v494f19991984 maven-plugin:3.19 metrics:4.1.6.2 mina-sshd-api-common:2.8.0-30.vf9df64641cb_d mina-sshd-api-core:2.8.0-30.vf9df64641cb_d momentjs:1.1.1 okhttp-api:4.9.3-105.vb96869f8ac3a pam-auth:1.8 parameterized-trigger:2.45 pipeline-build-step:2.18 pipeline-github-lib:38.v445716eaedda pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:593.va_a_fc25d520e9 pipeline-input-step:449.v77f0e8b_845c4 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2114.v2654ca_721309 pipeline-model-definition:2.2114.v2654ca_721309 pipeline-model-extensions:2.2114.v2654ca_721309 pipeline-rest-api:2.24 pipeline-stage-step:293.v200037eefcd5 pipeline-stage-tags-metadata:2.2114.v2654ca_721309 pipeline-stage-view:2.24 plain-credentials:1.8 plugin-util-api:2.17.0 popper-api:1.16.1-3 popper2-api:2.11.5-2 prometheus:2.0.11 pubsub-light:1.16 resource-disposer:0.19 run-condition:1.5 scm-api:608.vfa_f971c5a_a_e9 script-security:1175.v4b_d517d6db_f0 snakeyaml-api:1.30.2-76.vc104f7ce9870 sse-gateway:1.25 ssh-credentials:291.v8211e4f8efb_c ssh-slaves:1.821.vd834f8a_c390e sshd:3.242.va_db_9da_b_26a_c3 structs:318.va_f3ccb_729b_71 timestamper:1.18 token-macro:293.v283932a_0a_b_49 trilead-api:1.67.vc3938a_35172f variant:1.4 windows-slaves:1.8.1 workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1188.v0016b_4f29881 workflow-basic-steps:980.v82219a_ed188e workflow-cps:2746.v0da_83a_332669 workflow-cps-global-lib:588.v576c103a_ff86 workflow-durable-task-step:1190.vc93d7d457042 workflow-job:1207.ve6191ff089f8 workflow-multibranch:716.vc692ae52371b workflow-scm-step:400.v6b_89a1317c9a workflow-step-api:625.vd896b_f445a_f8 workflow-support:833.va1c71061486b ws-cleanup:0.42

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux

Reproduction steps

  1. Configure Amazon EC2 Container Service Cloud plugin with Maximum Agents set to 1

Expected Results

Expected all jobs that use the same label and template to be queued and that they will be executed one by one.

Actual Results

Jobs do run in queue in case parent job is already started (agent is created for it and it's preparing to be executed on it).

Jobs do not run in queue if they are all in build queue. So for example if you trigger 3 different jobs to run at once, they will all be executed at once, instead of only one job and then the other ones one by one after it.

Anything else?

No response

marcus-lim-lrn commented 2 years ago

I just discovered this issue

tomcart90 commented 1 year ago

This is causing major problems for us at the moment. It feels like this behaviour regressed at some point.

tomcart90 commented 1 year ago

I've had a bit of a look at this today and wonder if the cause here is that the provisioning call here doesn't take into account the maxAgents value, it simply requests the number of agents required by the number of items in the queue. So while subsequent provisioning operations would be blocked by isAtLimit returning true the initial call would simply provision whatever was required.

tomcart90 commented 1 year ago

I've raised https://github.com/jenkinsci/amazon-ecs-plugin/pull/284 to address this.

Stericson commented 1 year ago

284 merged. Will be in next release.