search

jenkinsci / amazon-ecs-plugin

Amazon EC2 Container Service Plugin for Jenkins
https://plugins.jenkins.io/amazon-ecs
MIT License
192 stars 227 forks source link

On amazon-ecs-plugin version 1.46, jenkins cannot start any jenkins agent on aws ECS EC2 #308

Closed squalou closed 1 year ago

squalou commented 1 year ago

Jenkins and plugins versions report

see https://github.com/jenkinsci/amazon-ecs-plugin/issues/298

Jenkins: 2.375.2
OS: Linux - 4.14.296-222.539.amzn2.x86_64
---
ace-editor:1.1
amazon-ecr:1.114.vfd22430621f5
amazon-ecs:1.41
ant:481.v7b_09e538fcca
antisamy-markup-formatter:155.v795fb_8702324
apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61
authentication-tokens:1.4
aws-credentials:191.vcb_f183ce58b_9
aws-java-sdk:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-cloudformation:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-codebuild:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ec2:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ecr:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ecs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-efs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-elasticbeanstalk:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-iam:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-logs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-minimal:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-sns:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-sqs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ssm:1.12.287-357.vf82d85a_6eefd
bootstrap4-api:4.6.0-5
bootstrap5-api:5.2.1-3
bouncycastle-api:2.27
branch-api:2.1071.v1a_188a_562481
build-monitor-plugin:1.13+build.202205140447
build-timeout:1.28
build-token-root:151.va_e52fe3215fc
build-user-vars-plugin:1.9
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:1.8.1
cloudbees-folder:6.800.v71307ca_b_986b
command-launcher:90.v669d7ccb_7c31
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-27.vb_fa_3896786a_7
cors-filter:1.1
credentials:1214.v1de940103927
credentials-binding:523.vd859a_4b_122e6
cucumber-reports:5.7.4
dark-theme:262.v0202a_4c8fb_6a
data-tables-api:1.12.1-4
date-parameter:0.0.4
display-url-api:2.3.7
docker-commons:1.21
docker-workflow:563.vd5d2e5c4007f
durable-task:504.vb10d1ae5ba2f
echarts-api:5.4.0-1
email-ext:2.93
external-monitor-job:203.v683c09d993b_9
font-awesome-api:6.2.1-1
git:5.0.0
git-client:4.1.0
git-parameter:0.9.18
git-server:99.va_0826a_b_cdfa_d
github:1.36.1
github-api:1.303-400.v35c2d8258028
github-branch-source:1701.v00cc8184df93
gitlab-plugin:1.6.0
google-login:1.7
htmlpublisher:1.31
http_request:1.16
instance-identity:142.v04572ca_5b_265
ionicons-api:31.v4757b_6987003
jackson2-api:2.14.2-319.v37853346a_229
jakarta-activation-api:2.0.1-2
jakarta-mail-api:2.0.1-2
javadoc:226.v71211feb_e7e9
javax-activation-api:1.2.0-5
javax-mail-api:1.6.2-8
jaxb:2.3.7-1
jdk-tool:63.v62d2fd4b_4793
jersey2-api:2.38-1
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.16-1
job-dsl:1.81
jobcacher:356.v7b_a_3fb_10a_10d
jquery:1.12.4-1
jquery3-api:3.6.1-2
jsch:0.1.55.61.va_e9ee26616e7
junit:1166.va_436e268e972
ldap:659.v8ca_b_a_fe79fa_d
lockable-resources:1117.v157231b_03882
mailer:438.v02c7f0a_12fa_4
matrix-auth:3.1.6
matrix-project:785.v06b_7f47b_c631
mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a
mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a
okhttp-api:4.9.3-108.v0feda04578cf
pam-auth:1.10
permissive-script-security:0.7
pipeline-aws:1.43
pipeline-build-step:2.18
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:629.vb_5627b_ee2104
pipeline-input-step:466.v6d0a_5df34f81
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2118.v31fd5b_9944b_5
pipeline-model-definition:2.2118.v31fd5b_9944b_5
pipeline-model-extensions:2.2118.v31fd5b_9944b_5
pipeline-rest-api:2.30
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5
pipeline-stage-view:2.30
pipeline-utility-steps:2.15.0
pitmutation:1.0-18
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:2.20.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
resource-disposer:0.20
saml:4.385.v4dea_91565e9d
scm-api:631.v9143df5b_e4a_a
script-security:1229.v4880b_b_e905a_6
snakeyaml-api:1.33-90.v80dcb_3814d35
sonar:2.15
ssh-agent:295.v9ca_a_1c7cc3a_a_
ssh-credentials:305.v8f4381501156
ssh-slaves:2.854.v7fd446b_337c9
sshd:3.275.v9e17c10f2571
structs:324.va_f5d6774f3a_d
theme-manager:1.6
timestamper:1.21
token-macro:321.vd7cc1f2a_52c8
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
view-job-filters:2.3
violation-comments-to-gitlab:2.57
workflow-aggregator:590.v6a_d052e5a_a_b_5
workflow-api:1208.v0cc7c6e0da_9e
workflow-basic-steps:994.vd57e3ca_46d24
workflow-cps:3606.v0b_d8b_e512dcf
workflow-durable-task-step:1223.v7f1a_98a_8863e
workflow-job:1268.v6eb_e2ee1a_85a
workflow-multibranch:716.vc692a_e52371b_
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.44

What Operating System are you using (both controller, and any agents involved in the problem)?

linux x68_64

Reproduction steps

Try to start a new jenkins agent using ecs-cloud plugin. Configure properly Ecs agent templates form.

Expected Results

Agent gets provisioned

Actual Results

2023-01-31 20:50:26.035+0000 [id=28]    INFO    c.c.j.plugins.amazonecs.ECSCloud#provision: Will provision factory-agent-jean-5cnq0, for label: agent-jean
2023-01-31 20:50:26.036+0000 [id=28]    SEVERE  hudson.triggers.SafeTimerTask#run: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@28c2b03a failed
java.lang.IllegalArgumentException
    at hudson.slaves.NodeProvisioner$PlannedNode.<init>(NodeProvisioner.java:102)
    at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.provision(ECSCloud.java:292)
    at com.cloudbees.jenkins.plugins.amazonecs.ECSProvisioningStrategy.apply(ECSProvisioningStrategy.java:66)
    at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325)
    at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:94)
    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    at java.base/java.util.concurrent.Sc

Anything else?

rollback to 1.41 as a workaround

squalou commented 1 year ago

The exception seems to come from

if (displayName == null || future == null || numExecutors < 1)  throw new IllegalArgumentException();

from the code changes introduced in 1.45, could be numExecutors taht would not be set to its default value (1) after plugin migration ? No idea and no way to test it easily.

Stericson commented 1 year ago

1.47 is the latest version and addresses this.

squalou commented 1 year ago

I'll give it a try. I'm a bit surprised since I tried to upgrade all the plugins 2 days ago, so the 1.47 should have been pulled. Anyway, will try asap and see what happens.

squalou commented 1 year ago

Hi, this is what I get after mifration 1.41->1.47

I manually set 1 there instead of zero and it works.

(Maybe something went wrong at my first attempt causing this situation now, anyway it's fixed on my setup, and hard to reproduce probably)

image

morfien101 commented 1 year ago

I have just bumped into the same issue, going from 1.41 -> 1.47. Updating the executors to 1 fixed the issue.